Sep 08 15:12:38.064568 ip-10-0-134-180 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 15:12:38.461795 ip-10-0-134-180 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:12:38.461795 ip-10-0-134-180 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 15:12:38.461795 ip-10-0-134-180 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:12:38.461795 ip-10-0-134-180 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 15:12:38.461795 ip-10-0-134-180 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 15:12:38.462997 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.462906    2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 15:12:38.467057 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467042    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.467057 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467057    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467061    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467065    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467068    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467071    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467074    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467077    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467082    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467086    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467089    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467092    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467094    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467097    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467100    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467103    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467106    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467108    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467111    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467114    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467116    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.467120 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467122    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467126    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467129    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467132    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467135    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467137    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467140    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467143    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467145    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467148    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467151    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467154    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467157    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467159    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467164    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467166    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467171    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467174    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467178    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.467623 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467181    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467184    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467187    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467189    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467192    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467194    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467197    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467199    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467202    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467205    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467207    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467210    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467212    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467215    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467219    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467222    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467225    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467228    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467231    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.468086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467234    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467236    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467239    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467242    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467244    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467254    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467256    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467259    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467263    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467266    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467269    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467272    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467274    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467277    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467280    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467282    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467285    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467287    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467300    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467303    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.468567 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467306    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467310    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467313    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467317    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467320    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467324    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467327    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467330    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467333    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467336    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467339    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467341    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467344    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467347    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467350    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467353    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467355    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467358    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467361    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.467363    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.469049 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468056    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468063    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468066    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468070    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468072    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468075    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468078    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468081    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468083    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468088    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468092    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468095    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468099    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468102    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468105    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468107    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468110    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468113    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468115    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.469592 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468119    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468122    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468124    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468127    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468130    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468134    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468138    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468142    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468146    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468149    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468152    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468154    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468157    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468160    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468163    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468165    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468168    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468171    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468173    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.470064 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468176    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468179    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468181    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468184    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468187    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468189    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468192    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468195    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468197    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468202    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468204    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468208    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468211    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468216    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468220    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468222    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468225    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468228    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468231    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.470552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468233    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468236    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468239    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468241    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468245    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468247    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468250    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468253    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468256    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468258    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468261    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468264    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468266    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468269    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468271    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468274    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468276    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468279    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468282    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468285    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.471022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468288    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468305    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468309    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468312    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468315    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468318    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468320    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468323    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468327    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468329    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468332    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468335    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468338    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468341    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468343    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468346    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468349    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468352    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468354    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468357    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.471610 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468360    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468362    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468716    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468720    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468723    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468726    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468729    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468732    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468735    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468738    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468741    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468744    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468747    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468750    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468753    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468756    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468759    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468762    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468765    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.472104 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468767    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468772    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468775    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468778    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468781    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468784    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468786    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468789    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468792    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468795    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468798    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468800    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468803    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468806    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468808    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468811    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468814    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468816    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468819    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468821    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.472576 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468824    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468827    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468829    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468832    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468835    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468838    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468840    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468843    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468846    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468850    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468852    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468856    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468860    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468863    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468865    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468868    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468871    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468874    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468877    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.473063 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468880    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468883    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468885    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468888    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468891    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468894    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468897    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468900    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468902    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468905    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468907    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468910    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468913    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468915    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468918    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468921    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468923    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468926    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468929    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468932    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.473538 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468935    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468938    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468940    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468944    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468946    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468949    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468952    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468954    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468957    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468960    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468962    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468965    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468967    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468970    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468973    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468975    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468978    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468980    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468983    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468986    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.474031 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468988    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468994    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.468996    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469064    2568 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469097    2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469105    2568 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469109    2568 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469114    2568 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469117    2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469121    2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469126    2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469129    2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469132    2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469135    2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469139    2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469142    2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469145    2568 flags.go:64] FLAG: --cgroup-root=""
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469148    2568 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469151    2568 flags.go:64] FLAG: --client-ca-file=""
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469154    2568 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469157    2568 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469161    2568 flags.go:64] FLAG: --cluster-domain=""
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469164    2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469167    2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 15:12:38.474605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469170    2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469173    2568 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469177    2568 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469180    2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469183    2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469186    2568 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469189    2568 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469192    2568 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469195    2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469199    2568 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469202    2568 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469206    2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469209    2568 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469212    2568 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469216    2568 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469219    2568 flags.go:64] FLAG: --enable-server="true"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469222    2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469227    2568 flags.go:64] FLAG: --event-burst="100"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469230    2568 flags.go:64] FLAG: --event-qps="50"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469233    2568 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469236    2568 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469239    2568 flags.go:64] FLAG: --eviction-hard=""
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469242    2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469245    2568 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469248    2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 15:12:38.475193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469252    2568 flags.go:64] FLAG: --eviction-soft=""
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469255    2568 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469258    2568 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469261    2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469264    2568 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469267    2568 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469270    2568 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469273    2568 flags.go:64] FLAG: --feature-gates=""
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469276    2568 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469279    2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469282    2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469286    2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469302    2568 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469305    2568 flags.go:64] FLAG: --help="false"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469308    2568 flags.go:64] FLAG: --hostname-override="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469311    2568 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469316    2568 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469319    2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469322    2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469326    2568 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469330    2568 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469333    2568 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469336    2568 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 15:12:38.475830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469339    2568 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469342    2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469345    2568 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469348    2568 flags.go:64] FLAG: --kube-reserved=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469351    2568 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469354    2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469357    2568 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469360    2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469363    2568 flags.go:64] FLAG: --lock-file=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469366    2568 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469369    2568 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469372    2568 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469378    2568 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469381    2568 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469383    2568 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469386    2568 flags.go:64] FLAG: --logging-format="text"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469389    2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469393    2568 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469396    2568 flags.go:64] FLAG: --manifest-url=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469399    2568 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469403    2568 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469406    2568 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469410    2568 flags.go:64] FLAG: --max-pods="110"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469413    2568 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469416    2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 15:12:38.476439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469420    2568 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469423    2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469426    2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469429    2568 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469432    2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469442    2568 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469445    2568 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469448    2568 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469451    2568 flags.go:64] FLAG: --pod-cidr=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469454    2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469459    2568 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469462    2568 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469466    2568 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469469    2568 flags.go:64] FLAG: --port="10250"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469472    2568 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469474    2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c219ff2d80bfe01e"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469477    2568 flags.go:64] FLAG: --qos-reserved=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469480    2568 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469484    2568 flags.go:64] FLAG: --register-node="true"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469487    2568 flags.go:64] FLAG: --register-with-taints=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469490    2568 flags.go:64] FLAG: --registry-burst="10"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469493    2568 flags.go:64] FLAG: --registry-qps="5"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469496    2568 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469499    2568 flags.go:64] FLAG: --reserved-memory=""
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469503    2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 15:12:38.477051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469506    2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469509    2568 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469512    2568 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469515    2568 flags.go:64] FLAG: --runonce="false"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469517    2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469520    2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469523    2568 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469527    2568 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469530    2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469533    2568 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469538    2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469541    2568 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469544    2568 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469547    2568 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469550    2568 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469553    2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469556    2568 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469559    2568 flags.go:64] FLAG: --system-cgroups=""
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469562    2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469568    2568 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469571    2568 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469573    2568 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469577    2568 flags.go:64] FLAG: --tls-min-version=""
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469580    2568 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469583    2568 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 15:12:38.477688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469586    2568 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469589    2568 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469593    2568 flags.go:64] FLAG: --v="2"
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469597    2568 flags.go:64] FLAG: --version="false"
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469601    2568 flags.go:64] FLAG: --vmodule=""
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469605    2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469608    2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469713    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469716    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469720    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469723    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469726    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469729    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469731    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469735    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469737    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469740    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469744    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469746    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469749    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469752    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.478353 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469755    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469758    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469760    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469763    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469766    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469768    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469771    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469774    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469776    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469779    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469781    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469784    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469787    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469790    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469793    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469797    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469800    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469803    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469805    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.478873 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469808    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469810    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469814    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469818    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469821    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469824    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469827    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469830    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469833    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469837    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469839    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469842    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469845    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469847    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469850    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469853    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469855    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469858    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469860    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469864    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.479414 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469866    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469869    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469871    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469875    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469879    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469882    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469885    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469888    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469891    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469894    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469896    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469899    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469901    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469904    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469906    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469909    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469912    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469915    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469917    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.479899 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469920    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469923    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469927    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469930    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469933    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469935    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469938    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469941    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469943    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469946    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469948    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469951    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469954    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469956    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469959    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469962    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469964    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469967    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469969    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469972    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.480378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469975    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469978    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469980    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469983    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469985    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469988    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.469990    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.469996    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.477827    2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.477843    2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.477862    2568 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.477868    2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477920    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477925    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477929    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.480874 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477932    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477935    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477938    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477941    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477944    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477947    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477949    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477952    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477954    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477959    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477963    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477966    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477969    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477972    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477975    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477978    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477980    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477984    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477986    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477989    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.481260 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477992    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477995    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.477997    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478000    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478004    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478008    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478012    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478015    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478018    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478021    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478024    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478027    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478030    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478033    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478036    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478039    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478042    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478045    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478048    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478051    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.481787 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478054    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478057    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478060    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478063    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478066    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478069    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478072    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478075    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478078    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478081    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478083    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478086    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478089    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478091    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478094    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478096    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478099    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478103    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478105    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478109    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.482272 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478112    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478114    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478117    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478120    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478122    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478125    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478127    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478130    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478132    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478135    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478138    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478140    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478143    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478146    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478148    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478151    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478154    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478156    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478159    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478161    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.482789 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478164    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478166    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478169    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478171    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478174    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478176    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478179    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478182    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478185    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478188    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478191    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478194    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478197    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478200    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478204    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.483270 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478206    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.478211    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478334    2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478340    2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478343    2568 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478346    2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478349    2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478352    2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478355    2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478358    2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478360    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478363    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478366    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478369    2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478372    2568 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 15:12:38.483646 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478374    2568 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478377    2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478379    2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478382    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478386    2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478389    2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478392    2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478395    2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478397    2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478400    2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478403    2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478406    2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478408    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478411    2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478414    2568 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478417    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478419    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478422    2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478424    2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 15:12:38.484018 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478427    2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478430    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478432    2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478442    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478445    2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478448    2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478452    2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478454    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478457    2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478459    2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478462    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478465    2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478468    2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478470    2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478473    2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478476    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478478    2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478481    2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478483    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478486    2568 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 15:12:38.484493 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478488    2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478492    2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478495    2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478498    2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478502    2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478505    2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478508    2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478511    2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478514    2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478518    2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478521    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478524    2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478526    2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478529    2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478532    2568 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478534    2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478537    2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478540    2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478542    2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 15:12:38.484981 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478545    2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478547    2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478550    2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478552    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478555    2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478557    2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478560    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478563    2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478566    2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478569    2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478571    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478574    2568 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478576    2568 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478579    2568 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478581    2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478584    2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478587    2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478590    2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478593    2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478595    2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 15:12:38.485452 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478598    2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478600    2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478603    2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478607    2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478610    2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478614    2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478617    2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:38.478620    2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.478625    2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.478727    2568 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.481030    2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.482165    2568 server.go:1019] "Starting client certificate rotation"
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.482258    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:12:38.485934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.482305    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 15:12:38.486255 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.484891    2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:12:38.492874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.492851    2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 15:12:38.504334 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.504313    2568 log.go:25] "Validated CRI v1 runtime API"
Sep 08 15:12:38.507618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.507601    2568 log.go:25] "Validated CRI v1 image API"
Sep 08 15:12:38.508390 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.508374    2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 15:12:38.512008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.511989    2568 fs.go:135] Filesystem UUIDs: map[17233d77-b209-4d31-9856-3c9ee75a363c:/dev/nvme0n1p4 746d71da-2fbc-4191-a132-458fe2e4f6d6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 08 15:12:38.512072 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.512008    2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 15:12:38.517162 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.517047    2568 manager.go:216] Machine: {Timestamp:2026-09-08 15:12:38.515663408 +0000 UTC m=+0.358460636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101752 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22bbbc66a3a761a21cbb144a6b5924 SystemUUID:ec22bbbc-66a3-a761-a21c-bb144a6b5924 BootID:e91be58c-3f4f-4f84-91bb-e6fbdf229aff Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:11:2d:f6:b2:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:11:2d:f6:b2:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:03:54:8f:ea:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 15:12:38.517162 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.517158    2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 15:12:38.517275 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.517264    2568 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 15:12:38.518133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.518118    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:12:38.519145 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.519119    2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 15:12:38.519275 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.519148    2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 15:12:38.519984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.519974    2568 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 15:12:38.520018 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.519985    2568 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 15:12:38.520018 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.519993    2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:12:38.520018 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.520007    2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 15:12:38.521994 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.521985    2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 15:12:38.526610 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526599    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:12:38.526717 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526708    2568 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 15:12:38.526755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526736    2568 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 15:12:38.526755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526745    2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 15:12:38.526755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526755    2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 15:12:38.526882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526764    2568 kubelet.go:398] "Adding apiserver pod source"
Sep 08 15:12:38.526882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.526773    2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 15:12:38.527105 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.527094    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:12:38.527142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.527112    2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 15:12:38.529465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.529438    2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 15:12:38.532426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.532413    2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 15:12:38.532469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.532444    2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 15:12:38.533333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533321    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 15:12:38.533368 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533340    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 15:12:38.533368 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533346    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 15:12:38.533368 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533352    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 15:12:38.533368 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533360    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 15:12:38.533368 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533366    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533385    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533394    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533405    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533411    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533421    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 15:12:38.533506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.533427    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 15:12:38.534214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.534204    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 15:12:38.534214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.534214    2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 15:12:38.538160 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.538139    2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-180.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 15:12:38.538366 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.538347    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 15:12:38.538468 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.538425    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 15:12:38.539032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.539017    2568 server.go:1268] "Started kubelet"
Sep 08 15:12:38.539123 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.539091    2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 15:12:38.539123 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.539103    2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 15:12:38.539274 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.539151    2568 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 15:12:38.539867 ip-10-0-134-180 systemd[1]: Started Kubernetes Kubelet.
Sep 08 15:12:38.540229 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.540212    2568 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 15:12:38.540606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.540587    2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 15:12:38.544288 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.544272    2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 15:12:38.547081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.547048    2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 15:12:38.547815 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.547689    2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 15:12:38.547885 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.547814    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:38.548367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548342    2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 15:12:38.548477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548342    2568 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 15:12:38.548554 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.547392    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354c63bb47  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.538984263 +0000 UTC m=+0.381781477,LastTimestamp:2026-09-08 15:12:38.538984263 +0000 UTC m=+0.381781477,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.548554 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548546    2568 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 15:12:38.548730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548616    2568 factory.go:55] Registering systemd factory
Sep 08 15:12:38.548730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548621    2568 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 15:12:38.548730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548631    2568 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 15:12:38.548730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548631    2568 factory.go:223] Registration of the systemd container factory successfully
Sep 08 15:12:38.548889 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548849    2568 factory.go:153] Registering CRI-O factory
Sep 08 15:12:38.548889 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548864    2568 factory.go:223] Registration of the crio container factory successfully
Sep 08 15:12:38.548970 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548954    2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 15:12:38.549016 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548972    2568 factory.go:103] Registering Raw factory
Sep 08 15:12:38.549016 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.548988    2568 manager.go:1196] Started watching for new ooms in manager
Sep 08 15:12:38.549331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.549318    2568 manager.go:349] Starting recovery of all containers
Sep 08 15:12:38.555536 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.555430    2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 15:12:38.555536 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.555515    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 15:12:38.560016 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.559999    2568 manager.go:354] Recovery completed
Sep 08 15:12:38.566647 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.566634    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.568742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.568717    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.568822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.568762    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.568822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.568775    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.569388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.569373    2568 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 15:12:38.569388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.569388    2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 15:12:38.569501 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.569405    2568 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 15:12:38.569501 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.569376    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.570647 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.570629    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.570716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.570671    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.570716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.570684    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.570907 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.570846    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e29d077  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568743031 +0000 UTC m=+0.411540252,LastTimestamp:2026-09-08 15:12:38.568743031 +0000 UTC m=+0.411540252,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.580910 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.580824    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e2a32a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568768167 +0000 UTC m=+0.411565384,LastTimestamp:2026-09-08 15:12:38.568768167 +0000 UTC m=+0.411565384,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.582861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.582843    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm9qh"
Sep 08 15:12:38.588444 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.588388    2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e2a6833  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568781875 +0000 UTC m=+0.411579092,LastTimestamp:2026-09-08 15:12:38.568781875 +0000 UTC m=+0.411579092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.590755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.590738    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tm9qh"
Sep 08 15:12:38.595794 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.595711    2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-180.ec2.internal.18d361354e29d077\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e29d077  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568743031 +0000 UTC m=+0.411540252,LastTimestamp:2026-09-08 15:12:38.570646612 +0000 UTC m=+0.413443830,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.604470    2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-180.ec2.internal.18d361354e2a32a7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e2a32a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568768167 +0000 UTC m=+0.411565384,LastTimestamp:2026-09-08 15:12:38.570677523 +0000 UTC m=+0.413474736,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.615209    2568 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-180.ec2.internal.18d361354e2a6833\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-180.ec2.internal.18d361354e2a6833  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-180.ec2.internal,UID:ip-10-0-134-180.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-180.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-180.ec2.internal,},FirstTimestamp:2026-09-08 15:12:38.568781875 +0000 UTC m=+0.411579092,LastTimestamp:2026-09-08 15:12:38.57068805 +0000 UTC m=+0.413485265,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-180.ec2.internal,}"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.648354    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.667793    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.668944    2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.668993    2568 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.669019    2568 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.669062    2568 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.680709    2568 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.710159    2568 policy_none.go:49] "None policy: Start"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.710174    2568 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 15:12:38.711071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.710184    2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 15:12:38.712320 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.712269    2568 policy_none.go:47] "Start"
Sep 08 15:12:38.741879 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.741865    2568 manager.go:337] "Starting Device Plugin manager"
Sep 08 15:12:38.741959 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.741906    2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 15:12:38.741959 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.741918    2568 server.go:87] "Starting device plugin registration server"
Sep 08 15:12:38.742202 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.742189    2568 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 15:12:38.742305 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.742205    2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 15:12:38.742366 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.742332    2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 15:12:38.742471 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.742449    2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 15:12:38.742471 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.742462    2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 15:12:38.742948 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.742929    2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 15:12:38.743016 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.742983    2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:38.761499 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.761481    2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-180.ec2.internal\" not found" node="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.769621 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.769588    2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"]
Sep 08 15:12:38.769709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.769665    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.771587 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.771571    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.771664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.771601    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.771664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.771612    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.773888 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.773876    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.773998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.773979    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.774036 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.774010    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.774648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.774630    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.774766 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.774658    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.774766 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.774669    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.775029 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.775015    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.775091 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.775040    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.775091 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.775054    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.776759 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.776747    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.776795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.776769    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.777440 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.777427    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.777499 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.777459    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.777499 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.777471    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.790817 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.790796    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-180.ec2.internal\" not found" node="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.794827 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.794810    2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-180.ec2.internal\" not found" node="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.842593 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.842572    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:38.843436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.843419    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:38.843528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.843447    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:38.843528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.843460    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:38.843528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.843486    2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.850664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.850646    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.850717 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.850670    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.850717 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.850687    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf1bd422ed1bc02488f8241ff80c9433-config\") pod \"kube-apiserver-proxy-ip-10-0-134-180.ec2.internal\" (UID: \"cf1bd422ed1bc02488f8241ff80c9433\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.851509 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.851496    2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.851547 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.851516    2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-180.ec2.internal\": node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:38.878941 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.878922    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:38.951371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.951445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951376    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.951445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951391    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.951445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf1bd422ed1bc02488f8241ff80c9433-config\") pod \"kube-apiserver-proxy-ip-10-0-134-180.ec2.internal\" (UID: \"cf1bd422ed1bc02488f8241ff80c9433\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.951541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf1bd422ed1bc02488f8241ff80c9433-config\") pod \"kube-apiserver-proxy-ip-10-0-134-180.ec2.internal\" (UID: \"cf1bd422ed1bc02488f8241ff80c9433\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.951541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:38.951443    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5007871134257076951b20479cadad03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal\" (UID: \"5007871134257076951b20479cadad03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:38.979446 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:38.979388    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.080102 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.080075    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.093234 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.093217    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:39.098541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.098023    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:39.181071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.180917    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.281553 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.281491    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.383731 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.383697    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.483233 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.483193    2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 15:12:39.484388 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.484356    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.585008 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.584936    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.639641 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.639616    2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:12:39.685570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.685540    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.710537 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.710507    2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:12:39.710648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.710525    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 15:07:39 +0000 UTC" deadline="2028-02-27 04:02:46.986555571 +0000 UTC"
Sep 08 15:12:39.710648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.710554    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12876h50m7.276004755s"
Sep 08 15:12:39.774654 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.774621    2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 15:12:39.775802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.775785    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 15:12:39.775876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.775816    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 15:12:39.775876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.775830    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeHasSufficientPID"
Sep 08 15:12:39.782387 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.782367    2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:12:39.786352 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.786332    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.886833 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.886796    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:39.962255 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:39.962217    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5007871134257076951b20479cadad03.slice/crio-b9e396b6efda7268e5f38d4eebc9824fd9e579be275d4b1b7ed662a90a5219b5 WatchSource:0}: Error finding container b9e396b6efda7268e5f38d4eebc9824fd9e579be275d4b1b7ed662a90a5219b5: Status 404 returned error can't find the container with id b9e396b6efda7268e5f38d4eebc9824fd9e579be275d4b1b7ed662a90a5219b5
Sep 08 15:12:39.966189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:39.966171    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:12:39.987760 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:39.987735    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:40.088236 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.088209    2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-180.ec2.internal\" not found"
Sep 08 15:12:40.120906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.120885    2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 15:12:40.149297 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.149276    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:40.174545 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.174526    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:12:40.175501 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.175489    2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal"
Sep 08 15:12:40.187669 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.187652    2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 15:12:40.530654 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.530575    2568 apiserver.go:52] "Watching apiserver"
Sep 08 15:12:40.536452 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.536421    2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 15:12:40.536876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.536854    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-nhpdt","openshift-network-operator/iptables-alerter-8wnzg","openshift-ovn-kubernetes/ovnkube-node-x865q","kube-system/global-pull-secret-syncer-5zq8s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb","openshift-cluster-node-tuning-operator/tuned-t2gwf","openshift-dns/node-resolver-vcdpg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal","openshift-multus/multus-additional-cni-plugins-54rt7","openshift-multus/multus-r9xd5","openshift-multus/network-metrics-daemon-bd7vx","kube-system/konnectivity-agent-gmsgv","kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal","openshift-image-registry/node-ca-44gr6"]
Sep 08 15:12:40.539528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.539505    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.541959 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.541934    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 15:12:40.541959 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.541948    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.542122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.541966    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m6gm8\""
Sep 08 15:12:40.542122 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.542015    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:40.542467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.542447    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.542568 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.542520    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.542677 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.542648    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 15:12:40.544930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.544528    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.547046 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547022    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.547265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547247    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 15:12:40.547488 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547465    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-79wnw\""
Sep 08 15:12:40.547594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547498    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 15:12:40.547594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547269    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 15:12:40.547714 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547355    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.547714 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547349    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.547954 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.547759    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 15:12:40.548614 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.548292    2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 15:12:40.549917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.549897    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.551280 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.551245    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 15:12:40.551408 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.551340    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jshnn\""
Sep 08 15:12:40.551593 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.551571    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 15:12:40.552154 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.552132    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.552505 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.552479    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.552608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.552523    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-shvbl\""
Sep 08 15:12:40.552608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.552574    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.552979 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.552921    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 15:12:40.554988 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.554966    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sg6pd\""
Sep 08 15:12:40.555761 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.555426    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.555761 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.555455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.555761 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.555511    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.558861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-etc-kubernetes\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.558861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558850    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.558984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558875    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-kubelet\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.558984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558904    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-ovn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.558984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558950    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-config\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.558991    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-sys-fs\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.559083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559017    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559045    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-conf-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559065    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wwkxw\""
Sep 08 15:12:40.559203 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559074    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-multus-certs\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559203 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559142    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a278541-4333-453b-998a-13f0fa707eae-ovn-node-metrics-cert\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559203 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559173    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njjqn\" (UniqueName: \"kubernetes.io/projected/8a278541-4333-453b-998a-13f0fa707eae-kube-api-access-njjqn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559299 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559220    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dbae73f-a055-4d17-ad74-579f71d20b66-agent-certs\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.559299 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559261    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-registration-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.559414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-netns\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559420    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 15:12:40.559458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559443    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.559529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559474    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-var-lib-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559499    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-log-socket\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559426    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 15:12:40.559598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-device-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.559698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559626    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-system-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559659    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-k8s-cni-cncf-io\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-kubelet\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559756    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-daemon-config\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.559901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559812    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-systemd-units\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.559901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559844    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560003 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559876    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560003 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.559936    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.560093 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560067    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-cni-binary-copy\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.560129 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560107    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-socket-dir-parent\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.560160 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560141    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-slash\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560212 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560194    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-netns\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560309 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-script-lib\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560381    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-socket-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.560455 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-hostroot\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.560504 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560450    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-cnibin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.560555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560517    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-multus\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.560555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560543    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560572    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-bin\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560701 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560653    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-netd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.560862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560841    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jf7\" (UniqueName: \"kubernetes.io/projected/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kube-api-access-92jf7\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.560991 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.560961    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmptq\" (UniqueName: \"kubernetes.io/projected/27e238d2-30e1-4ff7-b995-98d7173786f5-kube-api-access-zmptq\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.561065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561015    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-systemd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.561065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-etc-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.561161 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561109    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-node-log\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.561210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561149    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.561266 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561249    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-bin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.561340 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561281    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfffr\" (UniqueName: \"kubernetes.io/projected/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-kube-api-access-vfffr\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.561340 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561309    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-env-overrides\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.561450 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561387    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dbae73f-a055-4d17-ad74-579f71d20b66-konnectivity-ca\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.561740 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561715    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p9c76\""
Sep 08 15:12:40.561831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561765    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.561914 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.561892    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.562343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.562290    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.564757 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.564736    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:40.564866 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.564809    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:40.565669 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.565648    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 15:12:40.565923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.565905    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.566219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.566201    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vxps2\""
Sep 08 15:12:40.566529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.566506    2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 15:12:40.566626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.566512    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.567138 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.567118    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.567218 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.567179    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:40.569707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.569686    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.571947 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.571918    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 15:12:40.572361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.572337    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 15:12:40.572515 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.572494    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 15:12:40.572663 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.572644    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bnh8h\""
Sep 08 15:12:40.598058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.598040    2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9m4p"
Sep 08 15:12:40.609306 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.609286    2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c9m4p"
Sep 08 15:12:40.649977 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.649948    2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 15:12:40.662688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662651    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-cni-binary-copy\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.662688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662689    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-slash\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-netns\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662736    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-script-lib\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662763    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662773    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-slash\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662778    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-netns\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662792    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-lib-modules\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662833    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83812e0-af07-4572-8b7b-5f26596e433f-hosts-file\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662868    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-multus\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.662938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662919    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662933    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-multus\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662975    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jf7\" (UniqueName: \"kubernetes.io/projected/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kube-api-access-92jf7\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.662988    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-run-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663036    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-system-cni-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-tuned\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663123    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmptq\" (UniqueName: \"kubernetes.io/projected/27e238d2-30e1-4ff7-b995-98d7173786f5-kube-api-access-zmptq\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663160    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-systemd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663189    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-kubernetes\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663218    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b611c557-9379-4152-aca1-5abbd691f1f8-host-slash\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663232    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-systemd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663295    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-cni-binary-copy\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-bin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663350    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-cni-bin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663348    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfffr\" (UniqueName: \"kubernetes.io/projected/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-kube-api-access-vfffr\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663375    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-script-lib\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663395    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-env-overrides\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663421    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348f8ed7-b037-40a8-a964-500bd8d0520d-serviceca\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663439    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-etc-kubernetes\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663461    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-ovn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-sys-fs\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663505    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-etc-kubernetes\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663511    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348f8ed7-b037-40a8-a964-500bd8d0520d-host\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-dbus\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663563    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-ovn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663571    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-sys-fs\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663573    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-host\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663599    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qxr\" (UniqueName: \"kubernetes.io/projected/5d3ede11-ea1e-4040-91d6-bfc577b3e357-kube-api-access-k6qxr\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663622    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-systemd\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663675    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-k8s-cni-cncf-io\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.663696    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663710    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-k8s-cni-cncf-io\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663730    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-kubelet\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.663946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663755    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-var-lib-kubelet\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663783    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-systemd-units\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663807    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-systemd-units\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663818    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.663844    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:12:41.163821566 +0000 UTC m=+3.006618788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663857    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663900    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663925    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83812e0-af07-4572-8b7b-5f26596e433f-tmp-dir\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-socket-dir-parent\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.663975    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cnibin\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664002    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6c5d\" (UniqueName: \"kubernetes.io/projected/78af1c5d-0364-4e1a-a601-b219b5fd12c8-kube-api-access-z6c5d\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664044    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-modprobe-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664061    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysconfig\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664080    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-socket-dir-parent\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-run\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664144    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtkl\" (UniqueName: \"kubernetes.io/projected/b611c557-9379-4152-aca1-5abbd691f1f8-kube-api-access-gqtkl\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.664709 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664177    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-hostroot\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664223    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-hostroot\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664242    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-cnibin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664263    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-bin\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664278    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-netd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664318    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb4w\" (UniqueName: \"kubernetes.io/projected/348f8ed7-b037-40a8-a964-500bd8d0520d-kube-api-access-qlb4w\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-cnibin\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664337    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-netd\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664347    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7nt\" (UniqueName: \"kubernetes.io/projected/d83812e0-af07-4572-8b7b-5f26596e433f-kube-api-access-cg7nt\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-etc-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664403    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-node-log\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664415    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-etc-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664377    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-cni-bin\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664444    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-node-log\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664460    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-kubelet-config\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664482    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.665540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664507    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-conf\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664525    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664531    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-tmp\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dbae73f-a055-4d17-ad74-579f71d20b66-konnectivity-ca\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664591    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b611c557-9379-4152-aca1-5abbd691f1f8-iptables-alerter-script\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-kubelet\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-config\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664640    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-conf-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664676    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-multus-certs\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664677    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-host-kubelet\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664691    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a278541-4333-453b-998a-13f0fa707eae-ovn-node-metrics-cert\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664714    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njjqn\" (UniqueName: \"kubernetes.io/projected/8a278541-4333-453b-998a-13f0fa707eae-kube-api-access-njjqn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664729    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-conf-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664759    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-multus-certs\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664760    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-env-overrides\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664787    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dbae73f-a055-4d17-ad74-579f71d20b66-agent-certs\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.666082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664820    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-registration-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664850    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-netns\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664906    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-var-lib-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664880    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664952    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-log-socket\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664940    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-registration-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664980    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-host-run-netns\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.664983    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-device-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-var-lib-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665027    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-log-socket\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665039    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-device-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665064    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665065    2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665091    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665116    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-sys\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.666659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665135    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a278541-4333-453b-998a-13f0fa707eae-ovnkube-config\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665145    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-socket-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665172    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6dbae73f-a055-4d17-ad74-579f71d20b66-konnectivity-ca\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665239    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-system-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665264    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-socket-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665296    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-daemon-config\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665339    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665358    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665379    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-var-lib-kubelet\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665414    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27e238d2-30e1-4ff7-b995-98d7173786f5-system-cni-dir\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665427    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a278541-4333-453b-998a-13f0fa707eae-run-openvswitch\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665465    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.667174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.665765    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/27e238d2-30e1-4ff7-b995-98d7173786f5-multus-daemon-config\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.668543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.668521    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a278541-4333-453b-998a-13f0fa707eae-ovn-node-metrics-cert\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.668686 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.668668    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6dbae73f-a055-4d17-ad74-579f71d20b66-agent-certs\") pod \"konnectivity-agent-gmsgv\" (UID: \"6dbae73f-a055-4d17-ad74-579f71d20b66\") " pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.673162 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.673116    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal" event={"ID":"5007871134257076951b20479cadad03","Type":"ContainerStarted","Data":"b9e396b6efda7268e5f38d4eebc9824fd9e579be275d4b1b7ed662a90a5219b5"}
Sep 08 15:12:40.678721 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.678692    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jf7\" (UniqueName: \"kubernetes.io/projected/efe8ebe8-9656-4446-8ee5-1698e14ee98a-kube-api-access-92jf7\") pod \"aws-ebs-csi-driver-node-8wkbb\" (UID: \"efe8ebe8-9656-4446-8ee5-1698e14ee98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.678977 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.678942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njjqn\" (UniqueName: \"kubernetes.io/projected/8a278541-4333-453b-998a-13f0fa707eae-kube-api-access-njjqn\") pod \"ovnkube-node-x865q\" (UID: \"8a278541-4333-453b-998a-13f0fa707eae\") " pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.679071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.679046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmptq\" (UniqueName: \"kubernetes.io/projected/27e238d2-30e1-4ff7-b995-98d7173786f5-kube-api-access-zmptq\") pod \"multus-r9xd5\" (UID: \"27e238d2-30e1-4ff7-b995-98d7173786f5\") " pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.679126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.679090    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfffr\" (UniqueName: \"kubernetes.io/projected/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-kube-api-access-vfffr\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:40.766081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766048    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-lib-modules\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766118    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83812e0-af07-4572-8b7b-5f26596e433f-hosts-file\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766145    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-system-cni-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766203    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-system-cni-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766222    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d83812e0-af07-4572-8b7b-5f26596e433f-hosts-file\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766238    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-tuned\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766277    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-lib-modules\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766280    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-kubernetes\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766336    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-kubernetes\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766350    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b611c557-9379-4152-aca1-5abbd691f1f8-host-slash\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766382    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348f8ed7-b037-40a8-a964-500bd8d0520d-serviceca\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766418    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348f8ed7-b037-40a8-a964-500bd8d0520d-host\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766454    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-dbus\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-host\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766497    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b611c557-9379-4152-aca1-5abbd691f1f8-host-slash\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766520    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qxr\" (UniqueName: \"kubernetes.io/projected/5d3ede11-ea1e-4040-91d6-bfc577b3e357-kube-api-access-k6qxr\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766554    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-dbus\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766569    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-systemd\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766604    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-host\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766614    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766620    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-systemd\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766558    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348f8ed7-b037-40a8-a964-500bd8d0520d-host\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766661    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83812e0-af07-4572-8b7b-5f26596e433f-tmp-dir\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766692    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cnibin\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766723    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6c5d\" (UniqueName: \"kubernetes.io/projected/78af1c5d-0364-4e1a-a601-b219b5fd12c8-kube-api-access-z6c5d\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766751    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-modprobe-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.766817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysconfig\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766779    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766798    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-run\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766823    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtkl\" (UniqueName: \"kubernetes.io/projected/b611c557-9379-4152-aca1-5abbd691f1f8-kube-api-access-gqtkl\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766851    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb4w\" (UniqueName: \"kubernetes.io/projected/348f8ed7-b037-40a8-a964-500bd8d0520d-kube-api-access-qlb4w\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766875    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7nt\" (UniqueName: \"kubernetes.io/projected/d83812e0-af07-4572-8b7b-5f26596e433f-kube-api-access-cg7nt\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766903    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-kubelet-config\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766900    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348f8ed7-b037-40a8-a964-500bd8d0520d-serviceca\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766913    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-modprobe-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766928    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-conf\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.766996    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83812e0-af07-4572-8b7b-5f26596e433f-tmp-dir\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767004    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8efd27-0158-47b3-9e36-0ee192c74c98-kubelet-config\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767010    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-tmp\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767043    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b611c557-9379-4152-aca1-5abbd691f1f8-iptables-alerter-script\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767058    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysconfig\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767091    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.767692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767122    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767142    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767197    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767216    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-conf\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767221    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-run\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767229    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-sys\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767275    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-var-lib-kubelet\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767319    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-var-lib-kubelet\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767004    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cnibin\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.767397    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767490    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-sysctl-d\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.767511    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:41.267489393 +0000 UTC m=+3.110286600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767585    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b611c557-9379-4152-aca1-5abbd691f1f8-iptables-alerter-script\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767587    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/78af1c5d-0364-4e1a-a601-b219b5fd12c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767629    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d3ede11-ea1e-4040-91d6-bfc577b3e357-sys\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.768569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.767735    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/78af1c5d-0364-4e1a-a601-b219b5fd12c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.769399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.769060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-etc-tuned\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.769468 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.769418    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3ede11-ea1e-4040-91d6-bfc577b3e357-tmp\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.774888 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.774867    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:40.775026 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.774895    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:40.775026 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.774909    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:40.775026 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:40.774984    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:41.274965625 +0000 UTC m=+3.117762846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:40.776478 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.776421    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtkl\" (UniqueName: \"kubernetes.io/projected/b611c557-9379-4152-aca1-5abbd691f1f8-kube-api-access-gqtkl\") pod \"iptables-alerter-8wnzg\" (UID: \"b611c557-9379-4152-aca1-5abbd691f1f8\") " pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.777239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.777148    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qxr\" (UniqueName: \"kubernetes.io/projected/5d3ede11-ea1e-4040-91d6-bfc577b3e357-kube-api-access-k6qxr\") pod \"tuned-t2gwf\" (UID: \"5d3ede11-ea1e-4040-91d6-bfc577b3e357\") " pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.777239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.777196    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7nt\" (UniqueName: \"kubernetes.io/projected/d83812e0-af07-4572-8b7b-5f26596e433f-kube-api-access-cg7nt\") pod \"node-resolver-vcdpg\" (UID: \"d83812e0-af07-4572-8b7b-5f26596e433f\") " pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.777434 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.777287    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb4w\" (UniqueName: \"kubernetes.io/projected/348f8ed7-b037-40a8-a964-500bd8d0520d-kube-api-access-qlb4w\") pod \"node-ca-44gr6\" (UID: \"348f8ed7-b037-40a8-a964-500bd8d0520d\") " pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.778222 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.778200    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6c5d\" (UniqueName: \"kubernetes.io/projected/78af1c5d-0364-4e1a-a601-b219b5fd12c8-kube-api-access-z6c5d\") pod \"multus-additional-cni-plugins-54rt7\" (UID: \"78af1c5d-0364-4e1a-a601-b219b5fd12c8\") " pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.851781 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.851706    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r9xd5"
Sep 08 15:12:40.860509 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.860485    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:12:40.864299 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.864272    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e238d2_30e1_4ff7_b995_98d7173786f5.slice/crio-133289f868bc7533a1f0c1d8b39e7016447f4e7c2a970be795ef64c1d3dec048 WatchSource:0}: Error finding container 133289f868bc7533a1f0c1d8b39e7016447f4e7c2a970be795ef64c1d3dec048: Status 404 returned error can't find the container with id 133289f868bc7533a1f0c1d8b39e7016447f4e7c2a970be795ef64c1d3dec048
Sep 08 15:12:40.870412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.870372    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:40.870745 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.870718    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a278541_4333_453b_998a_13f0fa707eae.slice/crio-2ebac7a2b1d07ef0a14cb004402dad273e0db62da9a0d6c6fbb55383d6007d33 WatchSource:0}: Error finding container 2ebac7a2b1d07ef0a14cb004402dad273e0db62da9a0d6c6fbb55383d6007d33: Status 404 returned error can't find the container with id 2ebac7a2b1d07ef0a14cb004402dad273e0db62da9a0d6c6fbb55383d6007d33
Sep 08 15:12:40.877505 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.877485    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb"
Sep 08 15:12:40.880552 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.880505    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbae73f_a055_4d17_ad74_579f71d20b66.slice/crio-081d47be961ba8ac5c71001a57b93d6f4b571151a8c8b3e197172c02083bc2d2 WatchSource:0}: Error finding container 081d47be961ba8ac5c71001a57b93d6f4b571151a8c8b3e197172c02083bc2d2: Status 404 returned error can't find the container with id 081d47be961ba8ac5c71001a57b93d6f4b571151a8c8b3e197172c02083bc2d2
Sep 08 15:12:40.884281 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.884263    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2gwf"
Sep 08 15:12:40.887768 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.887746    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe8ebe8_9656_4446_8ee5_1698e14ee98a.slice/crio-8191316d5f2797e29bc066fcbe8045c51f73eba70773bd51bfc1c17f773942c4 WatchSource:0}: Error finding container 8191316d5f2797e29bc066fcbe8045c51f73eba70773bd51bfc1c17f773942c4: Status 404 returned error can't find the container with id 8191316d5f2797e29bc066fcbe8045c51f73eba70773bd51bfc1c17f773942c4
Sep 08 15:12:40.892375 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.892352    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-54rt7"
Sep 08 15:12:40.897400 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.897373    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3ede11_ea1e_4040_91d6_bfc577b3e357.slice/crio-3fdc9f42aadf3a967387c2d5abcbdf6f20348d2b3cbb042e4f564463725f4dfd WatchSource:0}: Error finding container 3fdc9f42aadf3a967387c2d5abcbdf6f20348d2b3cbb042e4f564463725f4dfd: Status 404 returned error can't find the container with id 3fdc9f42aadf3a967387c2d5abcbdf6f20348d2b3cbb042e4f564463725f4dfd
Sep 08 15:12:40.898587 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.898568    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcdpg"
Sep 08 15:12:40.903658 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.903636    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78af1c5d_0364_4e1a_a601_b219b5fd12c8.slice/crio-70c36bf0aa0511a0723e875b0aeb300291a027bed3f022f8d0e3a31575aa7092 WatchSource:0}: Error finding container 70c36bf0aa0511a0723e875b0aeb300291a027bed3f022f8d0e3a31575aa7092: Status 404 returned error can't find the container with id 70c36bf0aa0511a0723e875b0aeb300291a027bed3f022f8d0e3a31575aa7092
Sep 08 15:12:40.906255 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.906236    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8wnzg"
Sep 08 15:12:40.909642 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.909617    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83812e0_af07_4572_8b7b_5f26596e433f.slice/crio-8f710f21ebeeefdf56b21fc49e2b7c051bff6a318fcef15e08ef8a50b1871f51 WatchSource:0}: Error finding container 8f710f21ebeeefdf56b21fc49e2b7c051bff6a318fcef15e08ef8a50b1871f51: Status 404 returned error can't find the container with id 8f710f21ebeeefdf56b21fc49e2b7c051bff6a318fcef15e08ef8a50b1871f51
Sep 08 15:12:40.912991 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:40.912959    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44gr6"
Sep 08 15:12:40.916141 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.916120    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb611c557_9379_4152_aca1_5abbd691f1f8.slice/crio-f8d762d9e3efe3f288d2f16e45422b0f36a825e4cd139be045cae9c3787fa7ae WatchSource:0}: Error finding container f8d762d9e3efe3f288d2f16e45422b0f36a825e4cd139be045cae9c3787fa7ae: Status 404 returned error can't find the container with id f8d762d9e3efe3f288d2f16e45422b0f36a825e4cd139be045cae9c3787fa7ae
Sep 08 15:12:40.923298 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:12:40.923275    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348f8ed7_b037_40a8_a964_500bd8d0520d.slice/crio-7a3739cc000ee8d2cb833748ccbdf564464bddd951f1f36b2fc3afc94f8c828f WatchSource:0}: Error finding container 7a3739cc000ee8d2cb833748ccbdf564464bddd951f1f36b2fc3afc94f8c828f: Status 404 returned error can't find the container with id 7a3739cc000ee8d2cb833748ccbdf564464bddd951f1f36b2fc3afc94f8c828f
Sep 08 15:12:41.170607 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.170575    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:41.170803 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.170735    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:41.170803 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.170801    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:12:42.170782876 +0000 UTC m=+4.013580082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:41.271682 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.271645    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:41.271852 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.271835    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:41.271929 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.271917    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:42.271894993 +0000 UTC m=+4.114692215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:41.372001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.371968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:41.372179 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.372160    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:41.372233 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.372185    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:41.372233 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.372195    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:41.372301 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.372254    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:42.372240492 +0000 UTC m=+4.215037695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:41.614009 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.613918    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:07:41 +0000 UTC" deadline="2028-02-15 14:07:55.735691671 +0000 UTC"
Sep 08 15:12:41.614009 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.613959    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12598h55m14.121738407s"
Sep 08 15:12:41.669681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.669651    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:41.669835 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.669789    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:41.669905 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.669867    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:41.669965 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:41.669937    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:41.677118 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.677067    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerStarted","Data":"70c36bf0aa0511a0723e875b0aeb300291a027bed3f022f8d0e3a31575aa7092"}
Sep 08 15:12:41.678976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.678944    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmsgv" event={"ID":"6dbae73f-a055-4d17-ad74-579f71d20b66","Type":"ContainerStarted","Data":"081d47be961ba8ac5c71001a57b93d6f4b571151a8c8b3e197172c02083bc2d2"}
Sep 08 15:12:41.684847 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.684816    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"2ebac7a2b1d07ef0a14cb004402dad273e0db62da9a0d6c6fbb55383d6007d33"}
Sep 08 15:12:41.686573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.686377    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8wnzg" event={"ID":"b611c557-9379-4152-aca1-5abbd691f1f8","Type":"ContainerStarted","Data":"f8d762d9e3efe3f288d2f16e45422b0f36a825e4cd139be045cae9c3787fa7ae"}
Sep 08 15:12:41.689523 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.689492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2gwf" event={"ID":"5d3ede11-ea1e-4040-91d6-bfc577b3e357","Type":"ContainerStarted","Data":"3fdc9f42aadf3a967387c2d5abcbdf6f20348d2b3cbb042e4f564463725f4dfd"}
Sep 08 15:12:41.691590 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.691562    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb" event={"ID":"efe8ebe8-9656-4446-8ee5-1698e14ee98a","Type":"ContainerStarted","Data":"8191316d5f2797e29bc066fcbe8045c51f73eba70773bd51bfc1c17f773942c4"}
Sep 08 15:12:41.695134 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.695097    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r9xd5" event={"ID":"27e238d2-30e1-4ff7-b995-98d7173786f5","Type":"ContainerStarted","Data":"133289f868bc7533a1f0c1d8b39e7016447f4e7c2a970be795ef64c1d3dec048"}
Sep 08 15:12:41.697324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.697297    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44gr6" event={"ID":"348f8ed7-b037-40a8-a964-500bd8d0520d","Type":"ContainerStarted","Data":"7a3739cc000ee8d2cb833748ccbdf564464bddd951f1f36b2fc3afc94f8c828f"}
Sep 08 15:12:41.710114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:41.710024    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcdpg" event={"ID":"d83812e0-af07-4572-8b7b-5f26596e433f","Type":"ContainerStarted","Data":"8f710f21ebeeefdf56b21fc49e2b7c051bff6a318fcef15e08ef8a50b1871f51"}
Sep 08 15:12:42.181137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.181094    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:42.181361 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.181320    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:42.181464 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.181414    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:12:44.181392391 +0000 UTC m=+6.024189611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:42.282311 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.282148    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:42.282519 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.282352    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:42.282519 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.282430    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:44.282408819 +0000 UTC m=+6.125206034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:42.382864 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.382825    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:42.383051 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.383012    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:42.383051 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.383037    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:42.383051 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.383051    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:42.383224 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.383120    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:44.383100955 +0000 UTC m=+6.225898164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:42.614536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.614464    2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 15:07:41 +0000 UTC" deadline="2028-05-04 15:02:43.906614664 +0000 UTC"
Sep 08 15:12:42.614536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.614535    2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14495h50m1.292084571s"
Sep 08 15:12:42.669831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.669799    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:42.670004 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:42.669948    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:42.716726 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.716685    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal" event={"ID":"cf1bd422ed1bc02488f8241ff80c9433","Type":"ContainerStarted","Data":"9078eccdac44a95d6172b87a8b3d7dcadf775c1ca16f6f2a39d08f706703721e"}
Sep 08 15:12:42.722092 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.722059    2568 generic.go:356] "Generic (PLEG): container finished" podID="5007871134257076951b20479cadad03" containerID="cec9459355944a10a65091e73a3f59f8db051c0b0080de8683899dbf6341eda8" exitCode=0
Sep 08 15:12:42.722242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:42.722108    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal" event={"ID":"5007871134257076951b20479cadad03","Type":"ContainerDied","Data":"cec9459355944a10a65091e73a3f59f8db051c0b0080de8683899dbf6341eda8"}
Sep 08 15:12:43.669775 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:43.669681    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:43.670254 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:43.669835    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:43.670254 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:43.669898    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:43.670254 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:43.670029    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:44.199962 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:44.199916    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:44.200160 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.200125    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:44.200225 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.200188    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:12:48.200171059 +0000 UTC m=+10.042968280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:44.301286 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:44.301202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:44.301487 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.301378    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:44.301487 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.301460    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:48.301436447 +0000 UTC m=+10.144233657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:44.402912 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:44.402230    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:44.402912 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.402455    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:44.402912 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.402480    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:44.402912 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.402493    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:44.402912 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.402554    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:48.402535999 +0000 UTC m=+10.245333205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:44.669629 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:44.669592    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:44.669812 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:44.669758    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:45.669820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:45.669720    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:45.669820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:45.669748    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:45.670454 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:45.669896    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:45.670454 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:45.669992    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:46.670518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:46.670451    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:46.671019 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:46.670601    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:47.669545 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:47.669512    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:47.669749 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:47.669512    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:47.669749 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:47.669664    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:47.669749 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:47.669727    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:48.237575 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:48.237535    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:48.238053 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.237704    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:48.238053 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.237772    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:12:56.237752447 +0000 UTC m=+18.080549666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:48.338614 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:48.338572    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:48.338796 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.338777    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:48.338858 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.338844    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:56.338825449 +0000 UTC m=+18.181622658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:48.439961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:48.439922    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:48.440151 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.440133    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:48.440218 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.440161    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:48.440218 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.440178    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:48.440346 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.440243    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:12:56.440223598 +0000 UTC m=+18.283020801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:48.670755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:48.670300    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:48.670755 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:48.670462    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:49.670326 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:49.670239    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:49.670326 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:49.670311    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:49.670891 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:49.670397    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:49.670891 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:49.670546    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:50.670314 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:50.670188    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:50.670423 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:50.670338    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:51.669626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.669591    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:51.669626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.669609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:51.669856 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:51.669716    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:51.669856 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:51.669806    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:51.743777 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.743738    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2gwf" event={"ID":"5d3ede11-ea1e-4040-91d6-bfc577b3e357","Type":"ContainerStarted","Data":"d8d64e9587a554d153d553458adcae19a4126e2ecbde0da0621dfa8e4141d650"}
Sep 08 15:12:51.745796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.745764    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb" event={"ID":"efe8ebe8-9656-4446-8ee5-1698e14ee98a","Type":"ContainerStarted","Data":"0dbe4f18841815e62cbd5267e8cc2d778f6799748aa4507735ec82bbd30893ba"}
Sep 08 15:12:51.747595 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.747567    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal" event={"ID":"cf1bd422ed1bc02488f8241ff80c9433","Type":"ContainerStarted","Data":"9fb0dd2ec1a009f8d7a27ba215ac14dd5015b7ca7ce64a0bd20c421d87e956a0"}
Sep 08 15:12:51.750473 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.750445    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal" event={"ID":"5007871134257076951b20479cadad03","Type":"ContainerStarted","Data":"509ac0ff27c5b422253a771eaa0eaf63371d2d80693056ccb2ba8414ee2ddba0"}
Sep 08 15:12:51.752144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.752112    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44gr6" event={"ID":"348f8ed7-b037-40a8-a964-500bd8d0520d","Type":"ContainerStarted","Data":"11c450f61b07135c390c10a48b39ebea040c64ac75766eaafe6440ac4f0a3085"}
Sep 08 15:12:51.753770 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.753743    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcdpg" event={"ID":"d83812e0-af07-4572-8b7b-5f26596e433f","Type":"ContainerStarted","Data":"be0d6a3274e2422bd6135520e4614d211cba8ace4b9941395e3499b8378fbd46"}
Sep 08 15:12:51.755431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.755407    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerStarted","Data":"7d2520db822170a3c694831f421ffa8d37a3e3ad0e7e0eccd8951aff6f942f43"}
Sep 08 15:12:51.758526 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.758499    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmsgv" event={"ID":"6dbae73f-a055-4d17-ad74-579f71d20b66","Type":"ContainerStarted","Data":"b5dc3d43cbf4670bfb031be601b51563d3e0e31a1312eaf1bb0648c45042d14c"}
Sep 08 15:12:51.764974 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.764933    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t2gwf" podStartSLOduration=2.9805550309999997 podStartE2EDuration="12.764919752s" totalImagesPullingTime="9.784364721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.76451038 +0000 UTC m=+13.607307613" watchObservedRunningTime="2026-09-08 15:12:51.764919752 +0000 UTC m=+13.607716986"
Sep 08 15:12:51.799559 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.799467    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-44gr6" podStartSLOduration=3.518618102 podStartE2EDuration="11.799450974s" totalImagesPullingTime="8.280832872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.799130257 +0000 UTC m=+13.641927480" watchObservedRunningTime="2026-09-08 15:12:51.799450974 +0000 UTC m=+13.642248200"
Sep 08 15:12:51.841990 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.841752    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmsgv" podStartSLOduration=4.519604054 podStartE2EDuration="12.841732912s" totalImagesPullingTime="8.322128858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.822846514 +0000 UTC m=+13.665643738" watchObservedRunningTime="2026-09-08 15:12:51.841732912 +0000 UTC m=+13.684530135"
Sep 08 15:12:51.842569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.842538    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vcdpg" podStartSLOduration=3.070482828 podStartE2EDuration="12.842524234s" totalImagesPullingTime="9.772041406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.842042934 +0000 UTC m=+13.684840140" watchObservedRunningTime="2026-09-08 15:12:51.842524234 +0000 UTC m=+13.685321459"
Sep 08 15:12:51.857284 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.857239    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-180.ec2.internal" podStartSLOduration=11.857223412 podStartE2EDuration="11.857223412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.856759271 +0000 UTC m=+13.699556493" watchObservedRunningTime="2026-09-08 15:12:51.857223412 +0000 UTC m=+13.700020638"
Sep 08 15:12:51.870192 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:51.870146    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-180.ec2.internal" podStartSLOduration=11.870129238 podStartE2EDuration="11.870129238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:51.870016667 +0000 UTC m=+13.712813890" watchObservedRunningTime="2026-09-08 15:12:51.870129238 +0000 UTC m=+13.712926465"
Sep 08 15:12:52.673423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.673392    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:52.673623 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:52.673530    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:52.704701 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.704667    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:52.705421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.705403    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:12:52.762975 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.762938    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="7d2520db822170a3c694831f421ffa8d37a3e3ad0e7e0eccd8951aff6f942f43" exitCode=0
Sep 08 15:12:52.763485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.763041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"7d2520db822170a3c694831f421ffa8d37a3e3ad0e7e0eccd8951aff6f942f43"}
Sep 08 15:12:52.765396 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:52.765274    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8wnzg" event={"ID":"b611c557-9379-4152-aca1-5abbd691f1f8","Type":"ContainerStarted","Data":"70e33e215987c20a5bd93e6a73dbac54ac6b203604b0dbb656b63abf13497519"}
Sep 08 15:12:53.669552 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:53.669506    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:53.669710 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:53.669513    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:53.669710 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:53.669625    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:53.669794 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:53.669742    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:53.767371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:53.767340    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:12:54.669864 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:54.669827    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:54.670056 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:54.669974    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:55.669867 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:55.669831    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:55.670511 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:55.669830    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:55.670511 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:55.669958    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:55.670511 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:55.670027    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:56.299652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:56.299616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:56.299825 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.299799    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:56.299897 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.299884    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.299862931 +0000 UTC m=+34.142660138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:12:56.401035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:56.400995    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:56.401189 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.401159    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:56.401244 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.401229    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.401212247 +0000 UTC m=+34.244009448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:12:56.501960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:56.501923    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:56.502149 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.502116    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 15:12:56.502149 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.502141    2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 15:12:56.502274 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.502156    2568 projected.go:196] Error preparing data for projected volume kube-api-access-9hlr5 for pod openshift-network-diagnostics/network-check-target-nhpdt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:56.502274 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.502213    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5 podName:277f4418-f085-494e-8af3-caaa08942d18 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.502199286 +0000 UTC m=+34.344996491 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9hlr5" (UniqueName: "kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5") pod "network-check-target-nhpdt" (UID: "277f4418-f085-494e-8af3-caaa08942d18") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 15:12:56.669861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:56.669827    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:56.670042 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:56.669943    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:57.670025 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:57.669991    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:57.670193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:57.669991    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:57.670193 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:57.670120    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:57.670609 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:57.670212    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:58.670718 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:58.670690    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:12:58.671158 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:58.670790    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:12:59.577077 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.576864    2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 15:12:59.669486 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.669459    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:12:59.669581 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.669491    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:12:59.669637 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:59.669579    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:12:59.669692 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:12:59.669676    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:12:59.757608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.757505    2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T15:12:59.577074407Z","UUID":"096e3381-4e69-4994-af43-459e46f1324a","Handler":null,"Name":"","Endpoint":""}
Sep 08 15:12:59.760603 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.760513    2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 15:12:59.760603 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.760547    2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 15:12:59.781031 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.781000    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb" event={"ID":"efe8ebe8-9656-4446-8ee5-1698e14ee98a","Type":"ContainerStarted","Data":"1aff3590c7e0ddeef8def645c6f8919bf2a901c63304227d61ab31ff357a673b"}
Sep 08 15:12:59.782696 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.782632    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r9xd5" event={"ID":"27e238d2-30e1-4ff7-b995-98d7173786f5","Type":"ContainerStarted","Data":"9557d54ffbd1eaf88a7148f3fcc84d863958d64cc19dac10620a666ec39601d0"}
Sep 08 15:12:59.784801 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.784782    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"dc48d2a23ce12fcd3cd2174e208902d1d8501c75ec9d0f7e1d1b9255f38e1e98"}
Sep 08 15:12:59.784866 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.784834    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"eb547b6a442df1620b147856e851f10feef81a9f7f6ab607e0fc6814f573c11a"}
Sep 08 15:12:59.784866 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.784864    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"f4d589800742637e562191de72656ac2c5643624bdb81389df60fdc2f8cec71f"}
Sep 08 15:12:59.784936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.784891    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"7fd07df0fe01bc693aa533ea801e70395a9436e1669ef08c5129242ac701c355"}
Sep 08 15:12:59.816649 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.816409    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8wnzg" podStartSLOduration=11.007882218 podStartE2EDuration="20.816373954s" totalImagesPullingTime="9.808491736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:52.808503752 +0000 UTC m=+14.651300977" watchObservedRunningTime="2026-09-08 15:12:59.816373954 +0000 UTC m=+21.659171178"
Sep 08 15:12:59.817408 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:12:59.817372    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r9xd5" podStartSLOduration=2.252252581 podStartE2EDuration="20.817356536s" totalImagesPullingTime="18.565103955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:59.81673191 +0000 UTC m=+21.659529115" watchObservedRunningTime="2026-09-08 15:12:59.817356536 +0000 UTC m=+21.660153761"
Sep 08 15:13:00.669514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:00.669483    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:00.669656 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:00.669591    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:00.790112 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:00.790080    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"4b887c480b7877804d31dbea5f9e5ecb73568933cd32c8eba79aa16dd4772bcd"}
Sep 08 15:13:00.790483 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:00.790148    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"110f6e2571055505adc33102c56075779a482a3796cab3a91092c33aaed67299"}
Sep 08 15:13:01.669886 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:01.669849    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:01.670070 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:01.669849    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:01.670070 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:01.669961    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:01.670070 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:01.670053    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:02.669900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:02.669864    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:02.670461 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:02.669993    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:02.798311 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:02.798267    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"123eeb7ee24ad6e79cebc76dc51c7af83fe81daad8634acf03e7fdea30b88050"}
Sep 08 15:13:03.669802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:03.669767    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:03.669990 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:03.669898    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:03.669990 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:03.669962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:03.670421 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:03.670073    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:04.670170 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.670139    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:04.671071 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:04.670258    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:04.804307 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.804267    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="8fd5d1bfddccded3e9b42bfdea57348b02c2b42b4c496cfb2d01362194fd7161" exitCode=0
Sep 08 15:13:04.804396 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.804332    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"8fd5d1bfddccded3e9b42bfdea57348b02c2b42b4c496cfb2d01362194fd7161"}
Sep 08 15:13:04.810910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.810880    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" event={"ID":"8a278541-4333-453b-998a-13f0fa707eae","Type":"ContainerStarted","Data":"f27be74f4295684b8187c6c9607d4f18f57303ab2f805b249bd341a493c3dd39"}
Sep 08 15:13:04.811160 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.811115    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:04.811269 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.811172    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:04.828375 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.828354    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:04.849033 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:04.848995    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x865q" podStartSLOduration=7.241784837 podStartE2EDuration="25.848982372s" totalImagesPullingTime="18.607197535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:04.848232124 +0000 UTC m=+26.691029348" watchObservedRunningTime="2026-09-08 15:13:04.848982372 +0000 UTC m=+26.691779603"
Sep 08 15:13:05.411964 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.411930    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:13:05.412424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.412406    2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 15:13:05.413037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.413011    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmsgv"
Sep 08 15:13:05.670063 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.669862    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:05.670209 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.669872    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:05.670209 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:05.670119    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:05.670567 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:05.670213    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:05.814865 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.814837    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb" event={"ID":"efe8ebe8-9656-4446-8ee5-1698e14ee98a","Type":"ContainerStarted","Data":"197246b5472bf1c94805ce80a33b222556b16b3584b718dd6cb7123480c84d80"}
Sep 08 15:13:05.817022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.816999    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="98080c3679a855dffa1166802e40d91c3b631ad6c5468e2739c30907a9b0c942" exitCode=0
Sep 08 15:13:05.817126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.817056    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"98080c3679a855dffa1166802e40d91c3b631ad6c5468e2739c30907a9b0c942"}
Sep 08 15:13:05.817729 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.817708    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:05.832454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.832431    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:05.842555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:05.842519    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8wkbb" podStartSLOduration=3.3240783990000002 podStartE2EDuration="26.84250509s" totalImagesPullingTime="23.518426691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:05.842001893 +0000 UTC m=+27.684799116" watchObservedRunningTime="2026-09-08 15:13:05.84250509 +0000 UTC m=+27.685302313"
Sep 08 15:13:06.166998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.166963    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zq8s"]
Sep 08 15:13:06.167168 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.167088    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:06.167240 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:06.167212    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:06.168418 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.168392    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhpdt"]
Sep 08 15:13:06.168541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.168480    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:06.168604 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:06.168577    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:06.169121 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.169087    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bd7vx"]
Sep 08 15:13:06.169229 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.169202    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:06.169356 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:06.169328    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:06.821687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.821652    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="036d2e0a84e37d1593d5411b511c1d0a07e6ae9abcff39065da38c5022c7be0d" exitCode=0
Sep 08 15:13:06.822086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:06.821742    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"036d2e0a84e37d1593d5411b511c1d0a07e6ae9abcff39065da38c5022c7be0d"}
Sep 08 15:13:07.670065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:07.670037    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:07.670220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:07.670074    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:07.670220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:07.670123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:07.670220 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:07.670138    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:07.670220 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:07.670202    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:07.670380 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:07.670254    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:09.670305 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:09.670231    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:09.670846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:09.670379    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:09.670846 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:09.670433    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:09.670846 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:09.670477    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:09.670846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:09.670537    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:09.670846 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:09.670613    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:11.669302 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:11.669252    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:11.669869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:11.669320    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:11.669869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:11.669454    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:11.669869 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:11.669464    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bd7vx" podUID="bf1498fb-f0b5-4d98-81d7-cf9235433a3f"
Sep 08 15:13:11.669869 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:11.669580    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhpdt" podUID="277f4418-f085-494e-8af3-caaa08942d18"
Sep 08 15:13:11.669869 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:11.669653    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zq8s" podUID="fa8efd27-0158-47b3-9e36-0ee192c74c98"
Sep 08 15:13:11.975821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:11.975743    2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-180.ec2.internal" event="NodeReady"
Sep 08 15:13:11.975958 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:11.975877    2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 15:13:12.017726 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.017693    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"]
Sep 08 15:13:12.059365 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.059332    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:13:12.059535 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.059399    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.063213 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.063183    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.063407 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.063247    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.063582 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.063559    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kbbcv\""
Sep 08 15:13:12.064424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.064007    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 08 15:13:12.064956 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.064936    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 08 15:13:12.084233 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.084146    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"]
Sep 08 15:13:12.084378 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.084283    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.088191 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.088149    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nlfb9\""
Sep 08 15:13:12.088608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.088588    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 15:13:12.088721 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.088594    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 15:13:12.089599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.089443    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 15:13:12.094047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.094028    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 15:13:12.110353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.110330    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"]
Sep 08 15:13:12.110462 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.110374    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"]
Sep 08 15:13:12.110529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.110492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.113920 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.113900    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.114022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.113900    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.114584 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.114555    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 15:13:12.115358 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.115102    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5xvv\""
Sep 08 15:13:12.115358 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.115214    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 15:13:12.130720 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.130692    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"]
Sep 08 15:13:12.130842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.130819    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"
Sep 08 15:13:12.133077 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.133056    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.133360 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.133339    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.133693 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.133673    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qprgv\""
Sep 08 15:13:12.148816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.148789    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"]
Sep 08 15:13:12.148923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.148907    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.150890 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.150868    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 15:13:12.150890 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.150886    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 15:13:12.151104 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.150871    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.151198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.151184    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.151256 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.151244    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5pb46\""
Sep 08 15:13:12.170754 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.170733    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fdg9x"]
Sep 08 15:13:12.170867 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.170850    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.173116 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.173094    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 15:13:12.173208 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.173115    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.173208 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.173099    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.173208 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.173133    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6lzbs\""
Sep 08 15:13:12.197014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.196973    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"]
Sep 08 15:13:12.197122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.197036    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fgn85"]
Sep 08 15:13:12.197200 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.197176    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.199454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.199435    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 15:13:12.199564 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.199490    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.199722 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.199700    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.199814 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.199770    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 15:13:12.199814 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.199776    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p4bpj\""
Sep 08 15:13:12.204965 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.204861    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 15:13:12.211904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.211882    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.211904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.211921    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aac3e49-399c-4f97-a4b0-9343e12bc985-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.212086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.211943    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.211970    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.211996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcmq\" (UniqueName: \"kubernetes.io/projected/d07b1e65-794c-4466-9cdc-3a2f5a796479-kube-api-access-5lcmq\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.212330 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212304    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aac3e49-399c-4f97-a4b0-9343e12bc985-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.212436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212347    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212374    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fpr\" (UniqueName: \"kubernetes.io/projected/8aac3e49-399c-4f97-a4b0-9343e12bc985-kube-api-access-97fpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.212436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212394    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676ml\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212423    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpfk\" (UniqueName: \"kubernetes.io/projected/6637b6b4-7c9a-46ae-a129-373cc09bd5d7-kube-api-access-xlpfk\") pod \"volume-data-source-validator-695c895bd6-8xksq\" (UID: \"6637b6b4-7c9a-46ae-a129-373cc09bd5d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"
Sep 08 15:13:12.212642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212538    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d07b1e65-794c-4466-9cdc-3a2f5a796479-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.212642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212556    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.212642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212621    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.212825 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.212644    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.213200 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.213035    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-845c498454-fdsb6"]
Sep 08 15:13:12.213200 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.213160    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.216250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.215859    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.216250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.215935    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.216250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.216118    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 15:13:12.216250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.216129    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cgql7\""
Sep 08 15:13:12.216473 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.216396    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 15:13:12.231472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.231423    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 15:13:12.235984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.235967    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:13:12.236076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.235998    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"]
Sep 08 15:13:12.236076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236020    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"]
Sep 08 15:13:12.236076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236042    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"]
Sep 08 15:13:12.236076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236064    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.236280 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236066    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fdg9x"]
Sep 08 15:13:12.236388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236339    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fgn85"]
Sep 08 15:13:12.236388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.236369    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rc2px"]
Sep 08 15:13:12.238422 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238407    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 15:13:12.238512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238411    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 15:13:12.238560 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238528    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.238821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238804    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 15:13:12.238821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238816    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 15:13:12.238940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238877    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d8rrw\""
Sep 08 15:13:12.238940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.238892    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.256869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.256849    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.256869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.256851    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-24dw5"]
Sep 08 15:13:12.262754 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.262381    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j59vx\""
Sep 08 15:13:12.262754 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.262609    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 15:13:12.263005 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.262987    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 15:13:12.272914 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.272897    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-njhks"]
Sep 08 15:13:12.273014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.273000    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.275323 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.275245    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 15:13:12.275586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.275569    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2qkj4\""
Sep 08 15:13:12.275820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.275803    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 15:13:12.288446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.288422    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nxwbl"]
Sep 08 15:13:12.288557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.288517    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.290804 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.290788    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.291001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.290988    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9nsl6\""
Sep 08 15:13:12.291117 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.290977    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.291117 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.291050    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 15:13:12.305686 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305670    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-845c498454-fdsb6"]
Sep 08 15:13:12.305765 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc2px"]
Sep 08 15:13:12.305765 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305712    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nxwbl"]
Sep 08 15:13:12.305926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305765    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl"
Sep 08 15:13:12.305926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305798    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-24dw5"]
Sep 08 15:13:12.305926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.305817    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-njhks"]
Sep 08 15:13:12.307971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.307953    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 15:13:12.308073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.307981    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mgv2d\""
Sep 08 15:13:12.308073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.308011    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 15:13:12.312902 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.312885    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d07b1e65-794c-4466-9cdc-3a2f5a796479-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.312978 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.312907    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.312978 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.312927    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.312978 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.312957    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313015    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313044    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313057    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313074    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.813057602 +0000 UTC m=+34.655854816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313097    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.813085715 +0000 UTC m=+34.655882933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:12.313103 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313095    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89520eb2-f313-47fa-abf2-61065322b60b-serving-cert\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313125    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313166    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313202    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313223    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125e8c76-09e0-466a-8518-6469c0b019eb-serving-cert\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313246    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hpt\" (UniqueName: \"kubernetes.io/projected/712806b0-1710-4dda-8d6a-69e59e3c1839-kube-api-access-p2hpt\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313266    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aac3e49-399c-4f97-a4b0-9343e12bc985-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.313427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313407    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313442    2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313441    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-snapshots\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313499    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fpr\" (UniqueName: \"kubernetes.io/projected/8aac3e49-399c-4f97-a4b0-9343e12bc985-kube-api-access-97fpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313529    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676ml\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313580    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313603    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d07b1e65-794c-4466-9cdc-3a2f5a796479-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313608    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpfk\" (UniqueName: \"kubernetes.io/projected/6637b6b4-7c9a-46ae-a129-373cc09bd5d7-kube-api-access-xlpfk\") pod \"volume-data-source-validator-695c895bd6-8xksq\" (UID: \"6637b6b4-7c9a-46ae-a129-373cc09bd5d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbql\" (UniqueName: \"kubernetes.io/projected/125e8c76-09e0-466a-8518-6469c0b019eb-kube-api-access-xlbql\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.313677    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs podName:bf1498fb-f0b5-4d98-81d7-cf9235433a3f nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.313660679 +0000 UTC m=+66.156457899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs") pod "network-metrics-daemon-bd7vx" (UID: "bf1498fb-f0b5-4d98-81d7-cf9235433a3f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 15:13:12.313795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.313896    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-tmp\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314009    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314035    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-trusted-ca\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314058    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-config\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314093    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314118    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712806b0-1710-4dda-8d6a-69e59e3c1839-config\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314132    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8aac3e49-399c-4f97-a4b0-9343e12bc985-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314145    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmjk\" (UniqueName: \"kubernetes.io/projected/3b4b5685-e380-48c1-a770-e5c40f87381c-kube-api-access-djmjk\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314195    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2w7t\" (UniqueName: \"kubernetes.io/projected/89520eb2-f313-47fa-abf2-61065322b60b-kube-api-access-c2w7t\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314228    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712806b0-1710-4dda-8d6a-69e59e3c1839-serving-cert\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314261    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aac3e49-399c-4f97-a4b0-9343e12bc985-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.314343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314317    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.314802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314347    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.314802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcmq\" (UniqueName: \"kubernetes.io/projected/d07b1e65-794c-4466-9cdc-3a2f5a796479-kube-api-access-5lcmq\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.314802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.314387    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.317370 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.317217    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8aac3e49-399c-4f97-a4b0-9343e12bc985-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.317445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.317237    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.317445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.317434    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.328367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.328340    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fpr\" (UniqueName: \"kubernetes.io/projected/8aac3e49-399c-4f97-a4b0-9343e12bc985-kube-api-access-97fpr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-fnjnx\" (UID: \"8aac3e49-399c-4f97-a4b0-9343e12bc985\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.329025 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.328974    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.329571 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.329552    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676ml\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.329671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.329617    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcmq\" (UniqueName: \"kubernetes.io/projected/d07b1e65-794c-4466-9cdc-3a2f5a796479-kube-api-access-5lcmq\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.329671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.329626    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpfk\" (UniqueName: \"kubernetes.io/projected/6637b6b4-7c9a-46ae-a129-373cc09bd5d7-kube-api-access-xlpfk\") pod \"volume-data-source-validator-695c895bd6-8xksq\" (UID: \"6637b6b4-7c9a-46ae-a129-373cc09bd5d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"
Sep 08 15:13:12.371745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.371723    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"
Sep 08 15:13:12.414713 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414687    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-tmp-dir\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.414820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414728    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:12.414820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414749    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9mk\" (UniqueName: \"kubernetes.io/projected/04141987-e53e-4748-9ef5-897cc30217f7-kube-api-access-5g9mk\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.414820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414774    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kjjc\" (UniqueName: \"kubernetes.io/projected/94affd38-2c89-4b17-a472-c744de720572-kube-api-access-4kjjc\") pod \"network-check-source-685456655f-nxwbl\" (UID: \"94affd38-2c89-4b17-a472-c744de720572\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl"
Sep 08 15:13:12.414820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414798    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/849406e6-066e-4188-9f1c-062fd9d6bd5a-nginx-conf\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.414961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414843    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.414961 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.414855    2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 15:13:12.414961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414881    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.414961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89520eb2-f313-47fa-abf2-61065322b60b-serving-cert\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.414961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125e8c76-09e0-466a-8518-6469c0b019eb-serving-cert\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415224 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.414967    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hpt\" (UniqueName: \"kubernetes.io/projected/712806b0-1710-4dda-8d6a-69e59e3c1839-kube-api-access-p2hpt\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.415457 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.415420    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret podName:fa8efd27-0158-47b3-9e36-0ee192c74c98 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.41497248 +0000 UTC m=+66.257769690 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret") pod "global-pull-secret-syncer-5zq8s" (UID: "fa8efd27-0158-47b3-9e36-0ee192c74c98") : object "kube-system"/"original-pull-secret" not registered
Sep 08 15:13:12.415556 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.415556 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415531    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-snapshots\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415564    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbql\" (UniqueName: \"kubernetes.io/projected/125e8c76-09e0-466a-8518-6469c0b019eb-kube-api-access-xlbql\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415579    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-default-certificate\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.415668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415588    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415603    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.415668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-tmp\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415758    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.415904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415783    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-trusted-ca\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.415904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415812    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-stats-auth\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415931    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-config-volume\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415961    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-config\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.415996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04141987-e53e-4748-9ef5-897cc30217f7-cert\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416018    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712806b0-1710-4dda-8d6a-69e59e3c1839-config\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416039    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmjk\" (UniqueName: \"kubernetes.io/projected/3b4b5685-e380-48c1-a770-e5c40f87381c-kube-api-access-djmjk\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.416065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416064    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2w7t\" (UniqueName: \"kubernetes.io/projected/89520eb2-f313-47fa-abf2-61065322b60b-kube-api-access-c2w7t\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.416391 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416092    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712806b0-1710-4dda-8d6a-69e59e3c1839-serving-cert\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.416391 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416117    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-snapshots\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.416391 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416117    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416617    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416654    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-trusted-ca\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416658    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjxdh\" (UniqueName: \"kubernetes.io/projected/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-kube-api-access-cjxdh\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416690    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sc8\" (UniqueName: \"kubernetes.io/projected/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-kube-api-access-x4sc8\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.416748    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:12.416796 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.416796    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:12.916780332 +0000 UTC m=+34.759577536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:12.417033 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.416979    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712806b0-1710-4dda-8d6a-69e59e3c1839-config\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.417143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.417121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/89520eb2-f313-47fa-abf2-61065322b60b-config\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.417210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.417184    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/125e8c76-09e0-466a-8518-6469c0b019eb-tmp\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.417753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.417726    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/125e8c76-09e0-466a-8518-6469c0b019eb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.417864 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.417845    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/89520eb2-f313-47fa-abf2-61065322b60b-serving-cert\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.417906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.417867    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/125e8c76-09e0-466a-8518-6469c0b019eb-serving-cert\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.418734 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.418719    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712806b0-1710-4dda-8d6a-69e59e3c1839-serving-cert\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.423930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.423907    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hpt\" (UniqueName: \"kubernetes.io/projected/712806b0-1710-4dda-8d6a-69e59e3c1839-kube-api-access-p2hpt\") pod \"service-ca-operator-859f94c4b-n9hvs\" (UID: \"712806b0-1710-4dda-8d6a-69e59e3c1839\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.425286 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.425263    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbql\" (UniqueName: \"kubernetes.io/projected/125e8c76-09e0-466a-8518-6469c0b019eb-kube-api-access-xlbql\") pod \"insights-operator-76bcd6ccf5-fdg9x\" (UID: \"125e8c76-09e0-466a-8518-6469c0b019eb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.425715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.425691    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2w7t\" (UniqueName: \"kubernetes.io/projected/89520eb2-f313-47fa-abf2-61065322b60b-kube-api-access-c2w7t\") pod \"console-operator-85b454874b-fgn85\" (UID: \"89520eb2-f313-47fa-abf2-61065322b60b\") " pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.426282 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.426265    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmjk\" (UniqueName: \"kubernetes.io/projected/3b4b5685-e380-48c1-a770-e5c40f87381c-kube-api-access-djmjk\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.440711 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.440693    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"
Sep 08 15:13:12.457573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.457556    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"
Sep 08 15:13:12.508109 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.508062    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x"
Sep 08 15:13:12.517398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517377    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.517493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-default-certificate\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.517493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517441    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.517493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517459    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-stats-auth\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.517493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517476    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-config-volume\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517496    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517512    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04141987-e53e-4748-9ef5-897cc30217f7-cert\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517538    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.517568    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.517584    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.017559583 +0000 UTC m=+34.860356807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.517610    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.517628    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.017611928 +0000 UTC m=+34.860409130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.517655    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.017641187 +0000 UTC m=+34.860438412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:12.517681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517674    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjxdh\" (UniqueName: \"kubernetes.io/projected/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-kube-api-access-cjxdh\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517702    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sc8\" (UniqueName: \"kubernetes.io/projected/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-kube-api-access-x4sc8\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517735    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-tmp-dir\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517778    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9mk\" (UniqueName: \"kubernetes.io/projected/04141987-e53e-4748-9ef5-897cc30217f7-kube-api-access-5g9mk\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517807    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kjjc\" (UniqueName: \"kubernetes.io/projected/94affd38-2c89-4b17-a472-c744de720572-kube-api-access-4kjjc\") pod \"network-check-source-685456655f-nxwbl\" (UID: \"94affd38-2c89-4b17-a472-c744de720572\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517834    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/849406e6-066e-4188-9f1c-062fd9d6bd5a-nginx-conf\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.517871    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.518012    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.518051    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.018039422 +0000 UTC m=+34.860836636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:12.518108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.518104    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-config-volume\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.518540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.518134    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-tmp-dir\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.518955 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.518933    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/849406e6-066e-4188-9f1c-062fd9d6bd5a-nginx-conf\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:12.520031 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.520009    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04141987-e53e-4748-9ef5-897cc30217f7-cert\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.520335 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.520315    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlr5\" (UniqueName: \"kubernetes.io/projected/277f4418-f085-494e-8af3-caaa08942d18-kube-api-access-9hlr5\") pod \"network-check-target-nhpdt\" (UID: \"277f4418-f085-494e-8af3-caaa08942d18\") " pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:12.520434 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.520418    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-stats-auth\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.520517 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.520496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-default-certificate\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.522670 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.522654    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:12.527650 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.527630    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kjjc\" (UniqueName: \"kubernetes.io/projected/94affd38-2c89-4b17-a472-c744de720572-kube-api-access-4kjjc\") pod \"network-check-source-685456655f-nxwbl\" (UID: \"94affd38-2c89-4b17-a472-c744de720572\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl"
Sep 08 15:13:12.527737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.527682    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sc8\" (UniqueName: \"kubernetes.io/projected/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-kube-api-access-x4sc8\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:12.527913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.527896    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9mk\" (UniqueName: \"kubernetes.io/projected/04141987-e53e-4748-9ef5-897cc30217f7-kube-api-access-5g9mk\") pod \"ingress-canary-njhks\" (UID: \"04141987-e53e-4748-9ef5-897cc30217f7\") " pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.528251 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.528235    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjxdh\" (UniqueName: \"kubernetes.io/projected/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-kube-api-access-cjxdh\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:12.595573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.595553    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-njhks"
Sep 08 15:13:12.613575 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.613540    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl"
Sep 08 15:13:12.686856 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.686778    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx"]
Sep 08 15:13:12.822207 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.822064    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:12.822313 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.822244    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:12.822313 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.822260    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:12.822382 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.822339    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.822317721 +0000 UTC m=+35.665114938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:12.822478 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.822390    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:12.822478 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.822404    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:12.822478 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.822456    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.822439944 +0000 UTC m=+35.665237152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:12.838035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.838009    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerStarted","Data":"441c6b9f97dff90267fb7fe9d6c22e2f29099a518d899e48fc182930d1001d09"}
Sep 08 15:13:12.839230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.839206    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx" event={"ID":"8aac3e49-399c-4f97-a4b0-9343e12bc985","Type":"ContainerStarted","Data":"948c171f1a29a62859be72b1724a3e6638f9412bc102c040e9ce08b2c079baae"}
Sep 08 15:13:12.887540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.887503    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq"]
Sep 08 15:13:12.900152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.900129    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-njhks"]
Sep 08 15:13:12.903838 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.903816    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fdg9x"]
Sep 08 15:13:12.913831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.913809    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs"]
Sep 08 15:13:12.916806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.916788    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-fgn85"]
Sep 08 15:13:12.925975 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.925665    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:12.926077 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.926000    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:12.926125 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:12.926097    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:13.926066002 +0000 UTC m=+35.768863203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:12.928710 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:12.928411    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nxwbl"]
Sep 08 15:13:12.948133 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:12.948108    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04141987_e53e_4748_9ef5_897cc30217f7.slice/crio-65775d16c3b10358f0afabd5f48f90920f1d09e308c13040b9f8b7cc66bda1f1 WatchSource:0}: Error finding container 65775d16c3b10358f0afabd5f48f90920f1d09e308c13040b9f8b7cc66bda1f1: Status 404 returned error can't find the container with id 65775d16c3b10358f0afabd5f48f90920f1d09e308c13040b9f8b7cc66bda1f1
Sep 08 15:13:12.962455 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:12.962428    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125e8c76_09e0_466a_8518_6469c0b019eb.slice/crio-cff0cf1856d167f209c188e644a6afa1b5a818f7c73aa64bfb24a9d8d2be66f3 WatchSource:0}: Error finding container cff0cf1856d167f209c188e644a6afa1b5a818f7c73aa64bfb24a9d8d2be66f3: Status 404 returned error can't find the container with id cff0cf1856d167f209c188e644a6afa1b5a818f7c73aa64bfb24a9d8d2be66f3
Sep 08 15:13:12.962702 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:12.962674    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94affd38_2c89_4b17_a472_c744de720572.slice/crio-a64ab5d8cf12280940108e67172df137b26d2b58eac0c2d33b1b305af2deef80 WatchSource:0}: Error finding container a64ab5d8cf12280940108e67172df137b26d2b58eac0c2d33b1b305af2deef80: Status 404 returned error can't find the container with id a64ab5d8cf12280940108e67172df137b26d2b58eac0c2d33b1b305af2deef80
Sep 08 15:13:12.963983 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:12.963961    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89520eb2_f313_47fa_abf2_61065322b60b.slice/crio-7605cbd16c6fbee1915f24cf7c56a60c0f6f631b571424bed4950d5a52f43e2e WatchSource:0}: Error finding container 7605cbd16c6fbee1915f24cf7c56a60c0f6f631b571424bed4950d5a52f43e2e: Status 404 returned error can't find the container with id 7605cbd16c6fbee1915f24cf7c56a60c0f6f631b571424bed4950d5a52f43e2e
Sep 08 15:13:13.026744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.026660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:13.026744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.026713    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:13.026889 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.026819    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:14.026801795 +0000 UTC m=+35.869598995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:13.026889 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.026826    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:13.026889 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.026852    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:13.026889 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.026862    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:14.02685449 +0000 UTC m=+35.869651691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:13.027083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.026963    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:13.027083 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.026980    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:13.027083 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.027023    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:14.027010619 +0000 UTC m=+35.869807838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:13.027083 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.027046    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:13.027083 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.027096    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:14.027079995 +0000 UTC m=+35.869877199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:13.669312 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.669271    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:13.669312 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.669314    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:13.669582 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.669315    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:13.674333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.673921    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ws5f4\""
Sep 08 15:13:13.674333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.674089    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:13:13.674333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.674102    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:13:13.676107 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.675934    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dtm8b\""
Sep 08 15:13:13.685671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.685646    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:13.836423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.835576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:13.836423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.835630    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:13.836423 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.836164    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:13.836423 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.836184    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:13.836423 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.836243    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:15.836224498 +0000 UTC m=+37.679021717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:13.836911 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.836432    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:13.836911 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.836507    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:15.836493821 +0000 UTC m=+37.679291033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:13.849069 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.849037    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="441c6b9f97dff90267fb7fe9d6c22e2f29099a518d899e48fc182930d1001d09" exitCode=0
Sep 08 15:13:13.849198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.849129    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"441c6b9f97dff90267fb7fe9d6c22e2f29099a518d899e48fc182930d1001d09"}
Sep 08 15:13:13.852065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.852038    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl" event={"ID":"94affd38-2c89-4b17-a472-c744de720572","Type":"ContainerStarted","Data":"a64ab5d8cf12280940108e67172df137b26d2b58eac0c2d33b1b305af2deef80"}
Sep 08 15:13:13.854800 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.854775    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x" event={"ID":"125e8c76-09e0-466a-8518-6469c0b019eb","Type":"ContainerStarted","Data":"cff0cf1856d167f209c188e644a6afa1b5a818f7c73aa64bfb24a9d8d2be66f3"}
Sep 08 15:13:13.856363 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.856337    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs" event={"ID":"712806b0-1710-4dda-8d6a-69e59e3c1839","Type":"ContainerStarted","Data":"9ce017a5a252d43a411cf97c4fa9277784589bf26aa020f0a6f16966dc138ba4"}
Sep 08 15:13:13.858731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.858668    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq" event={"ID":"6637b6b4-7c9a-46ae-a129-373cc09bd5d7","Type":"ContainerStarted","Data":"4d2208cabeb2027bc33176f86bac83c0ab266dfc61f2fa34caea4f84661b115a"}
Sep 08 15:13:13.859989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.859951    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-njhks" event={"ID":"04141987-e53e-4748-9ef5-897cc30217f7","Type":"ContainerStarted","Data":"65775d16c3b10358f0afabd5f48f90920f1d09e308c13040b9f8b7cc66bda1f1"}
Sep 08 15:13:13.862516 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.862490    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fgn85" event={"ID":"89520eb2-f313-47fa-abf2-61065322b60b","Type":"ContainerStarted","Data":"7605cbd16c6fbee1915f24cf7c56a60c0f6f631b571424bed4950d5a52f43e2e"}
Sep 08 15:13:13.937044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.936973    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:13.937212 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.937196    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:13.937287 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:13.937262    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:15.937245045 +0000 UTC m=+37.780042264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:13.992631 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:13.992377    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhpdt"]
Sep 08 15:13:13.997861 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:13.997834    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277f4418_f085_494e_8af3_caaa08942d18.slice/crio-b4b30484dbe91e62813852bbea45b7b40e202efba41ac593d8319c8b3a557822 WatchSource:0}: Error finding container b4b30484dbe91e62813852bbea45b7b40e202efba41ac593d8319c8b3a557822: Status 404 returned error can't find the container with id b4b30484dbe91e62813852bbea45b7b40e202efba41ac593d8319c8b3a557822
Sep 08 15:13:14.038404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.038375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:14.038530 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.038449    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:14.038530 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038496    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:14.038530 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.038509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:14.038692 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038565    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:16.038544837 +0000 UTC m=+37.881342045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:14.038692 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.038628    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:14.038806 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038734    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:14.038806 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038769    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:16.038757122 +0000 UTC m=+37.881554330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:14.038917 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038885    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:16.038869539 +0000 UTC m=+37.881666758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:14.038974 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038957    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:14.039025 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:14.038993    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:16.038981848 +0000 UTC m=+37.881779068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:14.875028 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.874991    2568 generic.go:356] "Generic (PLEG): container finished" podID="78af1c5d-0364-4e1a-a601-b219b5fd12c8" containerID="b5222a699ebebe03b33e01b000b9216693a69bb0dc70ca15abce7417d0d44d3a" exitCode=0
Sep 08 15:13:14.875627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.875094    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerDied","Data":"b5222a699ebebe03b33e01b000b9216693a69bb0dc70ca15abce7417d0d44d3a"}
Sep 08 15:13:14.878032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:14.877983    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhpdt" event={"ID":"277f4418-f085-494e-8af3-caaa08942d18","Type":"ContainerStarted","Data":"b4b30484dbe91e62813852bbea45b7b40e202efba41ac593d8319c8b3a557822"}
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:15.860741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:15.860797    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.861043    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.861061    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.861096    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.861118    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:19.861101184 +0000 UTC m=+41.703898388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:15.861480 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.861162    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:19.861143861 +0000 UTC m=+41.703941064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:15.962043 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:15.961940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:15.962497 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.962153    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:15.962497 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:15.962266    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:19.962206886 +0000 UTC m=+41.805004091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:16.063090 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:16.063061    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:16.063237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:16.063121    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:16.063237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:16.063168    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:16.063381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:16.063261    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:16.063381 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063285    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:16.063381 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063366    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:20.063328748 +0000 UTC m=+41.906125975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:16.063521 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063388    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:20.063377732 +0000 UTC m=+41.906174939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:16.063521 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063404    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:16.063521 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063422    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:16.063521 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063467    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:20.063446407 +0000 UTC m=+41.906243608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:16.063521 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:16.063486    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:20.063479045 +0000 UTC m=+41.906276247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:19.898111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:19.898077    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:19.898128    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.898222    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.898275    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.898307    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.898311    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:27.89827357 +0000 UTC m=+49.741070777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:19.898570 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.898343    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:27.898331851 +0000 UTC m=+49.741129061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:19.999137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:19.999104    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:19.999355 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.999225    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:19.999355 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:19.999276    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:27.999264026 +0000 UTC m=+49.842061227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:20.099621 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:20.099592    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:20.099737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:20.099638    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:20.099737 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099721    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:20.099841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:20.099733    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:20.099841 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099767    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:28.099755499 +0000 UTC m=+49.942552700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:20.099841 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099813    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:20.099841 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099832    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:20.100046 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099859    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:28.099845228 +0000 UTC m=+49.942642451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:20.100046 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.099885    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:28.099869928 +0000 UTC m=+49.942667365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:20.100046 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:20.099944    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:20.100164 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:20.100067    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:28.100054836 +0000 UTC m=+49.942852040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:21.904202 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.903789    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-54rt7" event={"ID":"78af1c5d-0364-4e1a-a601-b219b5fd12c8","Type":"ContainerStarted","Data":"b92793ff620b2584272b27e48b5038357282b876620cf86ec172d71b0c35645a"}
Sep 08 15:13:21.905615 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.905578    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl" event={"ID":"94affd38-2c89-4b17-a472-c744de720572","Type":"ContainerStarted","Data":"73875b48ef2009f4086f9652e68abaa5a4d54bdfd7be24b3cb5d75e0de8b8cce"}
Sep 08 15:13:21.907259 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.907208    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x" event={"ID":"125e8c76-09e0-466a-8518-6469c0b019eb","Type":"ContainerStarted","Data":"e527e4f59a375b740a469f69cd4cb2d87704b3677159196df3ded3c61a07fd36"}
Sep 08 15:13:21.909267 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.909243    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhpdt" event={"ID":"277f4418-f085-494e-8af3-caaa08942d18","Type":"ContainerStarted","Data":"cf6b1f46415d19720f7b9e05bff529aec3ed857412954b02ce6a2a1432bb804e"}
Sep 08 15:13:21.909399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.909288    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:21.910919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.910889    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs" event={"ID":"712806b0-1710-4dda-8d6a-69e59e3c1839","Type":"ContainerStarted","Data":"496dd2ce4f37003585ade49c7c12ceb3a225a75192da9a1f70cbc23834d179c6"}
Sep 08 15:13:21.912814 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.912795    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq" event={"ID":"6637b6b4-7c9a-46ae-a129-373cc09bd5d7","Type":"ContainerStarted","Data":"5dc7594e7d0dfbad405aa7cec1b0c03a8774f8892b67d821eb243abcee9fa90b"}
Sep 08 15:13:21.914436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.914416    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-njhks" event={"ID":"04141987-e53e-4748-9ef5-897cc30217f7","Type":"ContainerStarted","Data":"861a9cca6993b8335ded0c666ebd10a461dde3b04fe8eb52445758af6835cba1"}
Sep 08 15:13:21.916151 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.916133    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/0.log"
Sep 08 15:13:21.916247 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.916171    2568 generic.go:356] "Generic (PLEG): container finished" podID="89520eb2-f313-47fa-abf2-61065322b60b" containerID="9442a63a4ef05b4ade45286e057049add7ccede3f8b4c33efc8cc4f40a07fe60" exitCode=255
Sep 08 15:13:21.916332 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.916254    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fgn85" event={"ID":"89520eb2-f313-47fa-abf2-61065322b60b","Type":"ContainerDied","Data":"9442a63a4ef05b4ade45286e057049add7ccede3f8b4c33efc8cc4f40a07fe60"}
Sep 08 15:13:21.916433 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.916418    2568 scope.go:117] "RemoveContainer" containerID="9442a63a4ef05b4ade45286e057049add7ccede3f8b4c33efc8cc4f40a07fe60"
Sep 08 15:13:21.917892 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.917869    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx" event={"ID":"8aac3e49-399c-4f97-a4b0-9343e12bc985","Type":"ContainerStarted","Data":"c8e1e136507b3b7c13d8affa4a7004f77d6ca038162a1fd627251af153545d23"}
Sep 08 15:13:21.928781 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.927670    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-54rt7" podStartSLOduration=14.120908884 podStartE2EDuration="42.927652206s" totalImagesPullingTime="28.806743322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:21.925095841 +0000 UTC m=+43.767893078" watchObservedRunningTime="2026-09-08 15:13:21.927652206 +0000 UTC m=+43.770449433"
Sep 08 15:13:21.960069 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.960014    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x" podStartSLOduration=10.970073792 podStartE2EDuration="18.95999384s" totalImagesPullingTime="7.989920048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:21.942933543 +0000 UTC m=+43.785730768" watchObservedRunningTime="2026-09-08 15:13:21.95999384 +0000 UTC m=+43.802791064"
Sep 08 15:13:21.961128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.961085    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs" podStartSLOduration=10.972014479 podStartE2EDuration="18.961070585s" totalImagesPullingTime="7.989056106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:21.958974395 +0000 UTC m=+43.801771619" watchObservedRunningTime="2026-09-08 15:13:21.961070585 +0000 UTC m=+43.803867809"
Sep 08 15:13:21.975214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:21.974974    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nhpdt" podStartSLOduration=36.017356496 podStartE2EDuration="42.974955923s" totalImagesPullingTime="6.957599427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:21.974176492 +0000 UTC m=+43.816973715" watchObservedRunningTime="2026-09-08 15:13:21.974955923 +0000 UTC m=+43.817753141"
Sep 08 15:13:22.010235 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.010167    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-njhks" podStartSLOduration=2.004834452 podStartE2EDuration="10.01014746s" totalImagesPullingTime="8.005313008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:22.010109383 +0000 UTC m=+43.852906608" watchObservedRunningTime="2026-09-08 15:13:22.01014746 +0000 UTC m=+43.852944682"
Sep 08 15:13:22.030427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.029646    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8xksq" podStartSLOduration=11.666074153 podStartE2EDuration="19.029626234s" totalImagesPullingTime="7.363552081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:22.028264007 +0000 UTC m=+43.871061225" watchObservedRunningTime="2026-09-08 15:13:22.029626234 +0000 UTC m=+43.872423459"
Sep 08 15:13:22.045912 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.045857    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-nxwbl" podStartSLOduration=11.05656875 podStartE2EDuration="19.04583679s" totalImagesPullingTime="7.98926804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:22.043867805 +0000 UTC m=+43.886665030" watchObservedRunningTime="2026-09-08 15:13:22.04583679 +0000 UTC m=+43.888634015"
Sep 08 15:13:22.523452 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.523412    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:22.523785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.523514    2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:22.702258 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.701707    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx" podStartSLOduration=11.098325951 podStartE2EDuration="19.701678957s" totalImagesPullingTime="8.603353006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:22.06972325 +0000 UTC m=+43.912520473" watchObservedRunningTime="2026-09-08 15:13:22.701678957 +0000 UTC m=+44.544476181"
Sep 08 15:13:22.703033 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.702714    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"]
Sep 08 15:13:22.726504 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.726470    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"]
Sep 08 15:13:22.726679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.726592    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"
Sep 08 15:13:22.729139 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.729119    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 15:13:22.729250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.729145    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 15:13:22.729318 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.729300    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wmcc2\""
Sep 08 15:13:22.830413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.830330    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2d5\" (UniqueName: \"kubernetes.io/projected/6216abf5-f88e-427e-b963-b9d97787533a-kube-api-access-jk2d5\") pod \"migrator-fdbb8fdff-ftz6x\" (UID: \"6216abf5-f88e-427e-b963-b9d97787533a\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"
Sep 08 15:13:22.925064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925043    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:13:22.925646 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925464    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/0.log"
Sep 08 15:13:22.925646 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925502    2568 generic.go:356] "Generic (PLEG): container finished" podID="89520eb2-f313-47fa-abf2-61065322b60b" containerID="d5a443c67f0e6eb4ac032d30da72b7e6c29761c438708026afc6a453312c1964" exitCode=255
Sep 08 15:13:22.925646 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925603    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fgn85" event={"ID":"89520eb2-f313-47fa-abf2-61065322b60b","Type":"ContainerDied","Data":"d5a443c67f0e6eb4ac032d30da72b7e6c29761c438708026afc6a453312c1964"}
Sep 08 15:13:22.925810 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925677    2568 scope.go:117] "RemoveContainer" containerID="9442a63a4ef05b4ade45286e057049add7ccede3f8b4c33efc8cc4f40a07fe60"
Sep 08 15:13:22.925849 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.925817    2568 scope.go:117] "RemoveContainer" containerID="d5a443c67f0e6eb4ac032d30da72b7e6c29761c438708026afc6a453312c1964"
Sep 08 15:13:22.926025 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:22.926001    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fgn85_openshift-console-operator(89520eb2-f313-47fa-abf2-61065322b60b)\"" pod="openshift-console-operator/console-operator-85b454874b-fgn85" podUID="89520eb2-f313-47fa-abf2-61065322b60b"
Sep 08 15:13:22.930817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.930793    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2d5\" (UniqueName: \"kubernetes.io/projected/6216abf5-f88e-427e-b963-b9d97787533a-kube-api-access-jk2d5\") pod \"migrator-fdbb8fdff-ftz6x\" (UID: \"6216abf5-f88e-427e-b963-b9d97787533a\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"
Sep 08 15:13:22.940373 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:22.940346    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2d5\" (UniqueName: \"kubernetes.io/projected/6216abf5-f88e-427e-b963-b9d97787533a-kube-api-access-jk2d5\") pod \"migrator-fdbb8fdff-ftz6x\" (UID: \"6216abf5-f88e-427e-b963-b9d97787533a\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"
Sep 08 15:13:23.038127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.038096    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"
Sep 08 15:13:23.073973 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.073941    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pzh5h"]
Sep 08 15:13:23.086408 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.086270    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.089145 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.089112    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzh5h"]
Sep 08 15:13:23.091059 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.091038    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 15:13:23.091194 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.091066    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 15:13:23.091194 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.091064    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xgtpp\""
Sep 08 15:13:23.133004 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.132806    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/d80a4af0-164c-48f4-a132-19de631f60f4-kube-api-access-pz8wp\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.133167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.133056    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.133167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.133104    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d80a4af0-164c-48f4-a132-19de631f60f4-crio-socket\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.133240 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.133215    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d80a4af0-164c-48f4-a132-19de631f60f4-data-volume\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.133424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.133372    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234432 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234395    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234464    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/d80a4af0-164c-48f4-a132-19de631f60f4-kube-api-access-pz8wp\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234510    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234594 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.234553    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:23.234594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234563    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d80a4af0-164c-48f4-a132-19de631f60f4-crio-socket\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234803 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.234618    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:23.734600945 +0000 UTC m=+45.577398146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:23.234803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234655    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d80a4af0-164c-48f4-a132-19de631f60f4-data-volume\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234803 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.234735    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:13:23.234803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.234784    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d80a4af0-164c-48f4-a132-19de631f60f4-crio-socket\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.234979 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.234796    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:23.734778634 +0000 UTC m=+45.577575850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : configmap "kube-rbac-proxy" not found
Sep 08 15:13:23.235049 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.235033    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d80a4af0-164c-48f4-a132-19de631f60f4-data-volume\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.244223 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.244196    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/d80a4af0-164c-48f4-a132-19de631f60f4-kube-api-access-pz8wp\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.328525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.328417    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x"]
Sep 08 15:13:23.331143 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:23.331111    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6216abf5_f88e_427e_b963_b9d97787533a.slice/crio-f255c9f6dd0135a41977f73ecd41d183dfcfbae7f612eeddfaa90220bb21c85f WatchSource:0}: Error finding container f255c9f6dd0135a41977f73ecd41d183dfcfbae7f612eeddfaa90220bb21c85f: Status 404 returned error can't find the container with id f255c9f6dd0135a41977f73ecd41d183dfcfbae7f612eeddfaa90220bb21c85f
Sep 08 15:13:23.739048 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.739014    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.739210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.739080    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:23.739311 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.739230    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:13:23.739311 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.739245    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:23.739412 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.739326    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:24.739304989 +0000 UTC m=+46.582102204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : configmap "kube-rbac-proxy" not found
Sep 08 15:13:23.739412 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.739352    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:24.739342134 +0000 UTC m=+46.582139348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:23.930907 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.930867    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x" event={"ID":"6216abf5-f88e-427e-b963-b9d97787533a","Type":"ContainerStarted","Data":"f255c9f6dd0135a41977f73ecd41d183dfcfbae7f612eeddfaa90220bb21c85f"}
Sep 08 15:13:23.932482 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.932465    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:13:23.932838 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:23.932820    2568 scope.go:117] "RemoveContainer" containerID="d5a443c67f0e6eb4ac032d30da72b7e6c29761c438708026afc6a453312c1964"
Sep 08 15:13:23.932980 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:23.932962    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-fgn85_openshift-console-operator(89520eb2-f313-47fa-abf2-61065322b60b)\"" pod="openshift-console-operator/console-operator-85b454874b-fgn85" podUID="89520eb2-f313-47fa-abf2-61065322b60b"
Sep 08 15:13:24.747834 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:24.747813    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:24.747914 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:24.747874    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:24.748018 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:24.747997    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:24.748089 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:24.748076    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:26.748055003 +0000 UTC m=+48.590852222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:24.748131 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:24.747999    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:13:24.748171 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:24.748149    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:26.748135897 +0000 UTC m=+48.590933099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : configmap "kube-rbac-proxy" not found
Sep 08 15:13:24.936695 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:24.936666    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x" event={"ID":"6216abf5-f88e-427e-b963-b9d97787533a","Type":"ContainerStarted","Data":"a15ffc46f6d89649d44df7e74582fe7312e4bbc9e4d9238bd7bd558c09a6c434"}
Sep 08 15:13:24.937077 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:24.936724    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x" event={"ID":"6216abf5-f88e-427e-b963-b9d97787533a","Type":"ContainerStarted","Data":"57a16aebec00969cc52232fb1f1f6c456dbfe3ea4aafbd5322f962f5b8b1ce43"}
Sep 08 15:13:24.952427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:24.952385    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-ftz6x" podStartSLOduration=1.5466201659999999 podStartE2EDuration="2.952370788s" totalImagesPullingTime="1.405750622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:24.951011926 +0000 UTC m=+46.793809150" watchObservedRunningTime="2026-09-08 15:13:24.952370788 +0000 UTC m=+46.795168068"
Sep 08 15:13:25.158352 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.158319    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hh8lx"]
Sep 08 15:13:25.176285 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.176263    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hh8lx"]
Sep 08 15:13:25.176428 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.176405    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.180041 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.180020    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 15:13:25.180146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.180061    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 15:13:25.180423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.180402    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 15:13:25.180423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.180419    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ccrpq\""
Sep 08 15:13:25.180559 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.180468    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 15:13:25.238611 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.238589    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcdpg_d83812e0-af07-4572-8b7b-5f26596e433f/dns-node-resolver/0.log"
Sep 08 15:13:25.252983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.252962    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djbp\" (UniqueName: \"kubernetes.io/projected/97c5b5d2-302c-475d-b55c-98d92dc87a31-kube-api-access-4djbp\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.253067 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.252998    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-key\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.253109 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.253069    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-cabundle\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.353644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.353611    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djbp\" (UniqueName: \"kubernetes.io/projected/97c5b5d2-302c-475d-b55c-98d92dc87a31-kube-api-access-4djbp\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.353775 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.353655    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-key\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.353775 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.353725    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-cabundle\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.354395 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.354374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-cabundle\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.356413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.356395    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97c5b5d2-302c-475d-b55c-98d92dc87a31-signing-key\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.361976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.361957    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djbp\" (UniqueName: \"kubernetes.io/projected/97c5b5d2-302c-475d-b55c-98d92dc87a31-kube-api-access-4djbp\") pod \"service-ca-75d99c484-hh8lx\" (UID: \"97c5b5d2-302c-475d-b55c-98d92dc87a31\") " pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.485586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.485519    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hh8lx"
Sep 08 15:13:25.778220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.778148    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hh8lx"]
Sep 08 15:13:25.781709 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:25.781685    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c5b5d2_302c_475d_b55c_98d92dc87a31.slice/crio-13118458764f0ed6932f77233d80b3590f87dedee23aab582de8a0ab742a05b6 WatchSource:0}: Error finding container 13118458764f0ed6932f77233d80b3590f87dedee23aab582de8a0ab742a05b6: Status 404 returned error can't find the container with id 13118458764f0ed6932f77233d80b3590f87dedee23aab582de8a0ab742a05b6
Sep 08 15:13:25.838152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.838129    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44gr6_348f8ed7-b037-40a8-a964-500bd8d0520d/node-ca/0.log"
Sep 08 15:13:25.942264 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.942232    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hh8lx" event={"ID":"97c5b5d2-302c-475d-b55c-98d92dc87a31","Type":"ContainerStarted","Data":"a113701d45d9d434732083c66405fc30605957d6e816f25634288b54163b36c1"}
Sep 08 15:13:25.942641 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.942305    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hh8lx" event={"ID":"97c5b5d2-302c-475d-b55c-98d92dc87a31","Type":"ContainerStarted","Data":"13118458764f0ed6932f77233d80b3590f87dedee23aab582de8a0ab742a05b6"}
Sep 08 15:13:25.962181 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:25.962129    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-hh8lx" podStartSLOduration=0.962109963 podStartE2EDuration="962.109963ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:25.960104143 +0000 UTC m=+47.802901367" watchObservedRunningTime="2026-09-08 15:13:25.962109963 +0000 UTC m=+47.804907188"
Sep 08 15:13:26.640587 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:26.640560    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-njhks_04141987-e53e-4748-9ef5-897cc30217f7/serve-healthcheck-canary/0.log"
Sep 08 15:13:26.766032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:26.765995    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:26.766222 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:26.766057    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:26.766222 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:26.766121    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:13:26.766222 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:26.766148    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:26.766222 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:26.766186    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:30.766170223 +0000 UTC m=+52.608967436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : configmap "kube-rbac-proxy" not found
Sep 08 15:13:26.766222 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:26.766205    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:30.766198773 +0000 UTC m=+52.608995974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:26.838015 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:26.837988    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ftz6x_6216abf5-f88e-427e-b963-b9d97787533a/migrator/0.log"
Sep 08 15:13:27.038133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:27.038061    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ftz6x_6216abf5-f88e-427e-b963-b9d97787533a/graceful-termination/0.log"
Sep 08 15:13:27.240957 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:27.240919    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fnjnx_8aac3e49-399c-4f97-a4b0-9343e12bc985/kube-storage-version-migrator-operator/0.log"
Sep 08 15:13:27.977311 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:27.977270    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:27.977341    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:27.977409    2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:27.977460    2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:27.977471    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls podName:d07b1e65-794c-4466-9cdc-3a2f5a796479 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:43.977452946 +0000 UTC m=+65.820250162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-l6msf" (UID: "d07b1e65-794c-4466-9cdc-3a2f5a796479") : secret "cluster-monitoring-operator-tls" not found
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:27.977472    2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c7b9d8c9-r456z: secret "image-registry-tls" not found
Sep 08 15:13:27.977524 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:27.977504    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls podName:19890a46-56dc-4ba3-af9a-0ce77e2c5f35 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:43.977495628 +0000 UTC m=+65.820292843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls") pod "image-registry-76c7b9d8c9-r456z" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35") : secret "image-registry-tls" not found
Sep 08 15:13:28.078656 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:28.078600    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:28.079120 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.078772    2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 15:13:28.079120 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.078853    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls podName:3b4b5685-e380-48c1-a770-e5c40f87381c nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.078833484 +0000 UTC m=+65.921630690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q2pwj" (UID: "3b4b5685-e380-48c1-a770-e5c40f87381c") : secret "samples-operator-tls" not found
Sep 08 15:13:28.179437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:28.179405    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:28.179628 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:28.179469    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:28.179628 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:28.179528    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:28.179628 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.179559    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.179541411 +0000 UTC m=+66.022338635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : configmap references non-existent config key: service-ca.crt
Sep 08 15:13:28.179628 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.179607    2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 15:13:28.179838 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.179645    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert podName:849406e6-066e-4188-9f1c-062fd9d6bd5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.17963473 +0000 UTC m=+66.022431946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-24dw5" (UID: "849406e6-066e-4188-9f1c-062fd9d6bd5a") : secret "networking-console-plugin-cert" not found
Sep 08 15:13:28.179838 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:28.179661    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:28.180153 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.180112    2568 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 15:13:28.180222 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.180211    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs podName:d127fd86-641c-4e8b-9dbb-a60dec4a0c5a nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.180192074 +0000 UTC m=+66.022989293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs") pod "router-default-845c498454-fdsb6" (UID: "d127fd86-641c-4e8b-9dbb-a60dec4a0c5a") : secret "router-metrics-certs-default" not found
Sep 08 15:13:28.180281 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.180215    2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 15:13:28.180281 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:28.180277    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls podName:b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:44.180253369 +0000 UTC m=+66.023050572 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls") pod "dns-default-rc2px" (UID: "b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49") : secret "dns-default-metrics-tls" not found
Sep 08 15:13:30.801192 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:30.801157    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:30.801656 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:30.801230    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:30.801656 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:30.801320    2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 15:13:30.801656 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:30.801332    2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:30.801656 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:30.801388    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:38.801373305 +0000 UTC m=+60.644170509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 15:13:30.801656 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:13:30.801402    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm podName:d80a4af0-164c-48f4-a132-19de631f60f4 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:38.801395756 +0000 UTC m=+60.644192956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pzh5h" (UID: "d80a4af0-164c-48f4-a132-19de631f60f4") : configmap "kube-rbac-proxy" not found
Sep 08 15:13:32.522871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.522837    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:32.523367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.522919    2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:32.523367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.523281    2568 scope.go:117] "RemoveContainer" containerID="d5a443c67f0e6eb4ac032d30da72b7e6c29761c438708026afc6a453312c1964"
Sep 08 15:13:32.972615 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.972587    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:13:32.972900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.972662    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-fgn85" event={"ID":"89520eb2-f313-47fa-abf2-61065322b60b","Type":"ContainerStarted","Data":"ecefafc4619807d099cff66295e2a035d6ce7f5b35cf8d15e853b39f8d2f375a"}
Sep 08 15:13:32.972900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.972879    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:32.989967 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:32.989916    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-fgn85" podStartSLOduration=22.030294379 podStartE2EDuration="29.989902019s" totalImagesPullingTime="7.95960764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:32.989269757 +0000 UTC m=+54.832066978" watchObservedRunningTime="2026-09-08 15:13:32.989902019 +0000 UTC m=+54.832699243"
Sep 08 15:13:33.123921 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:33.123891    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-fgn85"
Sep 08 15:13:37.833971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:37.833945    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x865q"
Sep 08 15:13:38.871335 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:38.871287    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:38.871738 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:38.871372    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:38.871878 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:38.871859    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d80a4af0-164c-48f4-a132-19de631f60f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:38.873492 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:38.873469    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d80a4af0-164c-48f4-a132-19de631f60f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pzh5h\" (UID: \"d80a4af0-164c-48f4-a132-19de631f60f4\") " pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:39.001972 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:39.001949    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xgtpp\""
Sep 08 15:13:39.008918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:39.008903    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pzh5h"
Sep 08 15:13:39.283186 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:39.283112    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pzh5h"]
Sep 08 15:13:39.286045 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:39.286020    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80a4af0_164c_48f4_a132_19de631f60f4.slice/crio-1a60e3b7ce1516da1aa67c8f27efb1d20e2714293754a4cfaebc2f3e32616cf6 WatchSource:0}: Error finding container 1a60e3b7ce1516da1aa67c8f27efb1d20e2714293754a4cfaebc2f3e32616cf6: Status 404 returned error can't find the container with id 1a60e3b7ce1516da1aa67c8f27efb1d20e2714293754a4cfaebc2f3e32616cf6
Sep 08 15:13:39.999870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:39.999832    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzh5h" event={"ID":"d80a4af0-164c-48f4-a132-19de631f60f4","Type":"ContainerStarted","Data":"5e1096b20a20cb76f9a9f80d8d22d1aa381abb165d445c0a4eb524ecbeafa7df"}
Sep 08 15:13:40.000338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:39.999911    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzh5h" event={"ID":"d80a4af0-164c-48f4-a132-19de631f60f4","Type":"ContainerStarted","Data":"1a60e3b7ce1516da1aa67c8f27efb1d20e2714293754a4cfaebc2f3e32616cf6"}
Sep 08 15:13:43.013476 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:43.013441    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzh5h" event={"ID":"d80a4af0-164c-48f4-a132-19de631f60f4","Type":"ContainerStarted","Data":"5e8f98b99defbb478d9ac51812ae246d313f920d290bf2eeadad71e38d150840"}
Sep 08 15:13:44.015510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.015480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:44.015988 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.015529    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:44.018310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.018237    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"image-registry-76c7b9d8c9-r456z\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") " pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:44.018438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.018421    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d07b1e65-794c-4466-9cdc-3a2f5a796479-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-l6msf\" (UID: \"d07b1e65-794c-4466-9cdc-3a2f5a796479\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:44.116358 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.116324    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:44.119122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.119098    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b4b5685-e380-48c1-a770-e5c40f87381c-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q2pwj\" (UID: \"3b4b5685-e380-48c1-a770-e5c40f87381c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:44.198535 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.198504    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nlfb9\""
Sep 08 15:13:44.206537 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.206512    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:44.217052 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.217018    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:44.217197 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.217107    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:44.217197 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.217146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:44.217333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.217200    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:44.217906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.217840    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-service-ca-bundle\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:44.219872 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.219834    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849406e6-066e-4188-9f1c-062fd9d6bd5a-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-24dw5\" (UID: \"849406e6-066e-4188-9f1c-062fd9d6bd5a\") " pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:44.219872 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.219849    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49-metrics-tls\") pod \"dns-default-rc2px\" (UID: \"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49\") " pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:44.220048 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.219981    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d127fd86-641c-4e8b-9dbb-a60dec4a0c5a-metrics-certs\") pod \"router-default-845c498454-fdsb6\" (UID: \"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a\") " pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:44.221725 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.221702    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m5xvv\""
Sep 08 15:13:44.229969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.229940    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"
Sep 08 15:13:44.283230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.283161    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6lzbs\""
Sep 08 15:13:44.291420 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.291396    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"
Sep 08 15:13:44.318376 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.318345    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:44.321258 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.321231    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 15:13:44.331400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.331374    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf1498fb-f0b5-4d98-81d7-cf9235433a3f-metrics-certs\") pod \"network-metrics-daemon-bd7vx\" (UID: \"bf1498fb-f0b5-4d98-81d7-cf9235433a3f\") " pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:44.346784 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.346767    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d8rrw\""
Sep 08 15:13:44.355566 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.355540    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:44.369737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.369704    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j59vx\""
Sep 08 15:13:44.378275 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.378240    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:44.383475 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.383237    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2qkj4\""
Sep 08 15:13:44.391621 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.391594    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5"
Sep 08 15:13:44.420344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.419721    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:44.422286 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.422209    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 15:13:44.433364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.433285    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8efd27-0158-47b3-9e36-0ee192c74c98-original-pull-secret\") pod \"global-pull-secret-syncer-5zq8s\" (UID: \"fa8efd27-0158-47b3-9e36-0ee192c74c98\") " pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:44.597831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.597751    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ws5f4\""
Sep 08 15:13:44.602835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.602810    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zq8s"
Sep 08 15:13:44.606627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.606609    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bd7vx"
Sep 08 15:13:44.612244 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.612226    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf"]
Sep 08 15:13:44.615929 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.615901    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07b1e65_794c_4466_9cdc_3a2f5a796479.slice/crio-c4322608c513a86ec712aaee44c54015045b996627051beb2f4c538b235b2e2f WatchSource:0}: Error finding container c4322608c513a86ec712aaee44c54015045b996627051beb2f4c538b235b2e2f: Status 404 returned error can't find the container with id c4322608c513a86ec712aaee44c54015045b996627051beb2f4c538b235b2e2f
Sep 08 15:13:44.618817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.618781    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj"]
Sep 08 15:13:44.626153 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.626119    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:13:44.631491 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.631454    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19890a46_56dc_4ba3_af9a_0ce77e2c5f35.slice/crio-1f8854d0488696bd36de8ca1ea1ba3688172cb7bfe45e7726785fa63d04e5d69 WatchSource:0}: Error finding container 1f8854d0488696bd36de8ca1ea1ba3688172cb7bfe45e7726785fa63d04e5d69: Status 404 returned error can't find the container with id 1f8854d0488696bd36de8ca1ea1ba3688172cb7bfe45e7726785fa63d04e5d69
Sep 08 15:13:44.690563 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.690540    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rc2px"]
Sep 08 15:13:44.695154 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.695126    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8eee15c_7b5d_48d2_b5ca_5b1c2d6fdf49.slice/crio-329d1e51642724c6325244067d7a0e6c1657937784ff6750ca7fb45d4a196b20 WatchSource:0}: Error finding container 329d1e51642724c6325244067d7a0e6c1657937784ff6750ca7fb45d4a196b20: Status 404 returned error can't find the container with id 329d1e51642724c6325244067d7a0e6c1657937784ff6750ca7fb45d4a196b20
Sep 08 15:13:44.707041 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.707016    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-24dw5"]
Sep 08 15:13:44.709662 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.709641    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849406e6_066e_4188_9f1c_062fd9d6bd5a.slice/crio-9d39de4944800169443b55f0dad71459ad45c75ad2d4ce76c265673a0667402f WatchSource:0}: Error finding container 9d39de4944800169443b55f0dad71459ad45c75ad2d4ce76c265673a0667402f: Status 404 returned error can't find the container with id 9d39de4944800169443b55f0dad71459ad45c75ad2d4ce76c265673a0667402f
Sep 08 15:13:44.709758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.709677    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-845c498454-fdsb6"]
Sep 08 15:13:44.712882 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.712862    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd127fd86_641c_4e8b_9dbb_a60dec4a0c5a.slice/crio-98007b1c46e974f2e806ae67e51955014c9342b8a88b04dcce384e79ccba266e WatchSource:0}: Error finding container 98007b1c46e974f2e806ae67e51955014c9342b8a88b04dcce384e79ccba266e: Status 404 returned error can't find the container with id 98007b1c46e974f2e806ae67e51955014c9342b8a88b04dcce384e79ccba266e
Sep 08 15:13:44.879874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.879845    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zq8s"]
Sep 08 15:13:44.882397 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.882358    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8efd27_0158_47b3_9e36_0ee192c74c98.slice/crio-19864121133e5a6fdf3f0bb6f46979d72fe24452c15e6a5d3501ed1969ca162b WatchSource:0}: Error finding container 19864121133e5a6fdf3f0bb6f46979d72fe24452c15e6a5d3501ed1969ca162b: Status 404 returned error can't find the container with id 19864121133e5a6fdf3f0bb6f46979d72fe24452c15e6a5d3501ed1969ca162b
Sep 08 15:13:44.884097 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:44.884074    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bd7vx"]
Sep 08 15:13:44.887694 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:44.887672    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1498fb_f0b5_4d98_81d7_cf9235433a3f.slice/crio-d8cf39a087431f17e10543f40c241939886357290645d2cd5b2da4807ee6aaf6 WatchSource:0}: Error finding container d8cf39a087431f17e10543f40c241939886357290645d2cd5b2da4807ee6aaf6: Status 404 returned error can't find the container with id d8cf39a087431f17e10543f40c241939886357290645d2cd5b2da4807ee6aaf6
Sep 08 15:13:45.022870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.022799    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-845c498454-fdsb6" event={"ID":"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a","Type":"ContainerStarted","Data":"5003acbb11363050e739a10ee7d6596cccbd16e357aa5ff1dd54527b0d2d8b7a"}
Sep 08 15:13:45.022870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.022870    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-845c498454-fdsb6" event={"ID":"d127fd86-641c-4e8b-9dbb-a60dec4a0c5a","Type":"ContainerStarted","Data":"98007b1c46e974f2e806ae67e51955014c9342b8a88b04dcce384e79ccba266e"}
Sep 08 15:13:45.025106 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.025076    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pzh5h" event={"ID":"d80a4af0-164c-48f4-a132-19de631f60f4","Type":"ContainerStarted","Data":"8a9b064b72f914937eb997b5d3549a952e33924ebf019c51032ddc0b9afe8413"}
Sep 08 15:13:45.026437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.026412    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bd7vx" event={"ID":"bf1498fb-f0b5-4d98-81d7-cf9235433a3f","Type":"ContainerStarted","Data":"d8cf39a087431f17e10543f40c241939886357290645d2cd5b2da4807ee6aaf6"}
Sep 08 15:13:45.027591 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.027569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj" event={"ID":"3b4b5685-e380-48c1-a770-e5c40f87381c","Type":"ContainerStarted","Data":"ddfb9a6e039a373ed20909423104f984e6b1701e0175049b6bea05b86e2baba5"}
Sep 08 15:13:45.028755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.028734    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc2px" event={"ID":"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49","Type":"ContainerStarted","Data":"329d1e51642724c6325244067d7a0e6c1657937784ff6750ca7fb45d4a196b20"}
Sep 08 15:13:45.029930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.029903    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf" event={"ID":"d07b1e65-794c-4466-9cdc-3a2f5a796479","Type":"ContainerStarted","Data":"c4322608c513a86ec712aaee44c54015045b996627051beb2f4c538b235b2e2f"}
Sep 08 15:13:45.031175 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.031148    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zq8s" event={"ID":"fa8efd27-0158-47b3-9e36-0ee192c74c98","Type":"ContainerStarted","Data":"19864121133e5a6fdf3f0bb6f46979d72fe24452c15e6a5d3501ed1969ca162b"}
Sep 08 15:13:45.032803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.032784    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" event={"ID":"19890a46-56dc-4ba3-af9a-0ce77e2c5f35","Type":"ContainerStarted","Data":"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"}
Sep 08 15:13:45.032881 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.032813    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" event={"ID":"19890a46-56dc-4ba3-af9a-0ce77e2c5f35","Type":"ContainerStarted","Data":"1f8854d0488696bd36de8ca1ea1ba3688172cb7bfe45e7726785fa63d04e5d69"}
Sep 08 15:13:45.032881 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.032834    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:13:45.034182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.034164    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5" event={"ID":"849406e6-066e-4188-9f1c-062fd9d6bd5a","Type":"ContainerStarted","Data":"9d39de4944800169443b55f0dad71459ad45c75ad2d4ce76c265673a0667402f"}
Sep 08 15:13:45.043333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.043253    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-845c498454-fdsb6" podStartSLOduration=42.043235843 podStartE2EDuration="42.043235843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:45.042140712 +0000 UTC m=+66.884937935" watchObservedRunningTime="2026-09-08 15:13:45.043235843 +0000 UTC m=+66.886033067"
Sep 08 15:13:45.062483 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.062440    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pzh5h" podStartSLOduration=17.085645157 podStartE2EDuration="22.062426632s" totalImagesPullingTime="4.976781475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:13:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:45.061811303 +0000 UTC m=+66.904608527" watchObservedRunningTime="2026-09-08 15:13:45.062426632 +0000 UTC m=+66.905223855"
Sep 08 15:13:45.082310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.082249    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" podStartSLOduration=65.082233325 podStartE2EDuration="1m5.082233325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:45.081124731 +0000 UTC m=+66.923921961" watchObservedRunningTime="2026-09-08 15:13:45.082233325 +0000 UTC m=+66.925030549"
Sep 08 15:13:45.363273 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.363183    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:45.366285 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:45.366258    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:46.042245 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:46.042179    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:46.044632 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:46.044608    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-845c498454-fdsb6"
Sep 08 15:13:47.651004 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.650969    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:13:47.654635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.654569    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.656934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.656873    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 15:13:47.656934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.656877    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 15:13:47.658083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.658049    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 15:13:47.658601 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.658579    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 15:13:47.659146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.659125    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 15:13:47.660993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.660964    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 15:13:47.661479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.661454    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 15:13:47.661568 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.661528    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8lf5\""
Sep 08 15:13:47.670536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.670511    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:13:47.750338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750268    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.750338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750322    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.750600 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750364    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.750600 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.750600 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750577    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvnz\" (UniqueName: \"kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.750732 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.750604    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.850968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvnz\" (UniqueName: \"kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851013    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851238 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851059    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851390    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851461    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851509    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.851900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.851876    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.852076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.852053    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.852458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.852429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.853679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.853651    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.854056 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.854031    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.858144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.858121    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvnz\" (UniqueName: \"kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz\") pod \"console-7bd97657dc-z7sx8\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") " pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:47.969141 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:47.969054    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:49.021134 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.021093    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:13:49.025575 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.025549    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.034098 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.034073    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 15:13:49.062361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.062336    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:13:49.161598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161620    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161650    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161698    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxvn\" (UniqueName: \"kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161780    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161828    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.161961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.161876    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.262943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.262905    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.262964    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263004    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263034    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263220    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263262    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxvn\" (UniqueName: \"kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.263935 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.263906    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.264064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.264030    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.264165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.264145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.264490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.264448    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.265778 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.265752    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.265923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.265904    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.271695 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.271646    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxvn\" (UniqueName: \"kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn\") pod \"console-fcbb9b8d-nl248\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") " pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.336638 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.336601    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:49.803731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.803645    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:13:49.808129 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:49.808094    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:13:49.824886 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:49.823761    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4588a581_7987_4d53_9641_8fbaec6c87e1.slice/crio-4ecfa123bc2457c9a06d5a46ec242803a8d791abbea05a462085f5d6a3d7cac8 WatchSource:0}: Error finding container 4ecfa123bc2457c9a06d5a46ec242803a8d791abbea05a462085f5d6a3d7cac8: Status 404 returned error can't find the container with id 4ecfa123bc2457c9a06d5a46ec242803a8d791abbea05a462085f5d6a3d7cac8
Sep 08 15:13:50.061733 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.061642    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5" event={"ID":"849406e6-066e-4188-9f1c-062fd9d6bd5a","Type":"ContainerStarted","Data":"188462cb50dad41fd0f659d5c02559224efb486ea7b7ef70894a3c1779a2fd12"}
Sep 08 15:13:50.063017 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.062991    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fcbb9b8d-nl248" event={"ID":"4588a581-7987-4d53-9641-8fbaec6c87e1","Type":"ContainerStarted","Data":"4ecfa123bc2457c9a06d5a46ec242803a8d791abbea05a462085f5d6a3d7cac8"}
Sep 08 15:13:50.064841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.064817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bd7vx" event={"ID":"bf1498fb-f0b5-4d98-81d7-cf9235433a3f","Type":"ContainerStarted","Data":"a4006fda0b40402366db0f15ebf2fb7d4087bb9baf51cd1b05822cfc87322e8f"}
Sep 08 15:13:50.064946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.064863    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bd7vx" event={"ID":"bf1498fb-f0b5-4d98-81d7-cf9235433a3f","Type":"ContainerStarted","Data":"3acdabab11926a0fb7c819ce9cf96a782fcddd571ab6c32fb39b8c07f6b38ddd"}
Sep 08 15:13:50.066824 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.066798    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj" event={"ID":"3b4b5685-e380-48c1-a770-e5c40f87381c","Type":"ContainerStarted","Data":"8e783b06688d60fdce1dcf4a0ee6b80370e7ee3db84c447efcc74bfb9ea38e3c"}
Sep 08 15:13:50.066926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.066848    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj" event={"ID":"3b4b5685-e380-48c1-a770-e5c40f87381c","Type":"ContainerStarted","Data":"edf44a791d7f23395c85a5c96b7664ef3bb6dd6650fb0211707fa11bde097b7a"}
Sep 08 15:13:50.068071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.068047    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd97657dc-z7sx8" event={"ID":"cef0bea7-3573-4183-b726-9fb4cadb3743","Type":"ContainerStarted","Data":"5f2359a53f72325abf03d18f780723e30c315def871e255d60117d999f89f4ef"}
Sep 08 15:13:50.069746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.069726    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc2px" event={"ID":"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49","Type":"ContainerStarted","Data":"08f552796da76eb56586d6e6c960f4ae78dc3e106e2fc3e167821ae2a77114d8"}
Sep 08 15:13:50.069855 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.069775    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rc2px"
Sep 08 15:13:50.069855 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.069799    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rc2px" event={"ID":"b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49","Type":"ContainerStarted","Data":"67a032c2801de2b95c1beeac78383f3b8a84291b33bdb3d1f2dce7201f9182d1"}
Sep 08 15:13:50.071324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.071303    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf" event={"ID":"d07b1e65-794c-4466-9cdc-3a2f5a796479","Type":"ContainerStarted","Data":"70d20e6ccd486b935f8f747e4d1970cba8fea4a0deff95db35fc07afc8583b3f"}
Sep 08 15:13:50.072793 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.072773    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zq8s" event={"ID":"fa8efd27-0158-47b3-9e36-0ee192c74c98","Type":"ContainerStarted","Data":"af864438026d7ad985d85879f9d5ff34ab347c11e7fde52d5cb992710e33ecf7"}
Sep 08 15:13:50.080517 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.080480    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-24dw5" podStartSLOduration=43.732034562 podStartE2EDuration="45.08046843s" totalImagesPullingTime="1.348433868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.079617122 +0000 UTC m=+71.922414349" watchObservedRunningTime="2026-09-08 15:13:50.08046843 +0000 UTC m=+71.923265653"
Sep 08 15:13:50.099990 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.099950    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rc2px" podStartSLOduration=33.280774077 podStartE2EDuration="38.099937193s" totalImagesPullingTime="4.819163116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.098898148 +0000 UTC m=+71.941695374" watchObservedRunningTime="2026-09-08 15:13:50.099937193 +0000 UTC m=+71.942734415"
Sep 08 15:13:50.118694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.118654    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q2pwj" podStartSLOduration=42.287879872 podStartE2EDuration="47.118643744s" totalImagesPullingTime="4.830763872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.117387449 +0000 UTC m=+71.960184671" watchObservedRunningTime="2026-09-08 15:13:50.118643744 +0000 UTC m=+71.961440966"
Sep 08 15:13:50.137198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.137155    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-l6msf" podStartSLOduration=42.243979708 podStartE2EDuration="47.137143121s" totalImagesPullingTime="4.893163413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.136156445 +0000 UTC m=+71.978953668" watchObservedRunningTime="2026-09-08 15:13:50.137143121 +0000 UTC m=+71.979940341"
Sep 08 15:13:50.165867 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.165821    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bd7vx" podStartSLOduration=66.535024122 podStartE2EDuration="1m11.165810619s" totalImagesPullingTime="4.630786497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.164995109 +0000 UTC m=+72.007792356" watchObservedRunningTime="2026-09-08 15:13:50.165810619 +0000 UTC m=+72.008607841"
Sep 08 15:13:50.186099 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.186051    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5zq8s" podStartSLOduration=65.493053135 podStartE2EDuration="1m10.186035453s" totalImagesPullingTime="4.692982318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:50.185541031 +0000 UTC m=+72.028338250" watchObservedRunningTime="2026-09-08 15:13:50.186035453 +0000 UTC m=+72.028832678"
Sep 08 15:13:50.230901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.230869    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"]
Sep 08 15:13:50.263846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.263817    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"]
Sep 08 15:13:50.263999 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.263944    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:50.266111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.266089    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 15:13:50.266231 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.266180    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-v4srt\""
Sep 08 15:13:50.373164 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.373132    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ff40d75-0c7e-41ea-bc47-9a6bd824b54c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zgbrv\" (UID: \"0ff40d75-0c7e-41ea-bc47-9a6bd824b54c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:50.473536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.473494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ff40d75-0c7e-41ea-bc47-9a6bd824b54c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zgbrv\" (UID: \"0ff40d75-0c7e-41ea-bc47-9a6bd824b54c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:50.475969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.475938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ff40d75-0c7e-41ea-bc47-9a6bd824b54c-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zgbrv\" (UID: \"0ff40d75-0c7e-41ea-bc47-9a6bd824b54c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:50.586352 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.586310    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:50.879264 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:50.879190    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"]
Sep 08 15:13:50.882831 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:13:50.882801    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff40d75_0c7e_41ea_bc47_9a6bd824b54c.slice/crio-9ab64bb38c944fa1de6eb4fd87bba3a0b361ab5826d167174035560cfe7791ef WatchSource:0}: Error finding container 9ab64bb38c944fa1de6eb4fd87bba3a0b361ab5826d167174035560cfe7791ef: Status 404 returned error can't find the container with id 9ab64bb38c944fa1de6eb4fd87bba3a0b361ab5826d167174035560cfe7791ef
Sep 08 15:13:51.081628 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:51.081027    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv" event={"ID":"0ff40d75-0c7e-41ea-bc47-9a6bd824b54c","Type":"ContainerStarted","Data":"9ab64bb38c944fa1de6eb4fd87bba3a0b361ab5826d167174035560cfe7791ef"}
Sep 08 15:13:52.928747 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:52.928715    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nhpdt"
Sep 08 15:13:55.102365 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.102330    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv" event={"ID":"0ff40d75-0c7e-41ea-bc47-9a6bd824b54c","Type":"ContainerStarted","Data":"db1b6ec6f502056694a01f9a1f07e5e21582c37ad6e5305b63453377529eb90e"}
Sep 08 15:13:55.102799 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.102475    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:55.104282 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.104258    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fcbb9b8d-nl248" event={"ID":"4588a581-7987-4d53-9641-8fbaec6c87e1","Type":"ContainerStarted","Data":"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"}
Sep 08 15:13:55.105929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.105905    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd97657dc-z7sx8" event={"ID":"cef0bea7-3573-4183-b726-9fb4cadb3743","Type":"ContainerStarted","Data":"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"}
Sep 08 15:13:55.107452 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.107425    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv"
Sep 08 15:13:55.120391 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.120350    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zgbrv" podStartSLOduration=2.011218548 podStartE2EDuration="5.120338161s" totalImagesPullingTime="3.109119613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:55.119399735 +0000 UTC m=+76.962196957" watchObservedRunningTime="2026-09-08 15:13:55.120338161 +0000 UTC m=+76.963135383"
Sep 08 15:13:55.156474 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.156429    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fcbb9b8d-nl248" podStartSLOduration=2.989779011 podStartE2EDuration="7.156418384s" totalImagesPullingTime="4.166639373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:55.1538669 +0000 UTC m=+76.996664122" watchObservedRunningTime="2026-09-08 15:13:55.156418384 +0000 UTC m=+76.999215609"
Sep 08 15:13:55.198404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:55.198361    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd97657dc-z7sx8" podStartSLOduration=4.02436712 podStartE2EDuration="8.198347338s" totalImagesPullingTime="4.173980218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:55.196925739 +0000 UTC m=+77.039722964" watchObservedRunningTime="2026-09-08 15:13:55.198347338 +0000 UTC m=+77.041144561"
Sep 08 15:13:57.969690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:57.969652    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:57.970059 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:57.969706    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:13:57.970989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:57.970961    2568 patch_prober.go:28] interesting pod/console-7bd97657dc-z7sx8 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Sep 08 15:13:57.971119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:57.971008    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bd97657dc-z7sx8" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Sep 08 15:13:59.339474 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.339419    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:59.339914 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.339651    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:13:59.339914 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.339848    2568 patch_prober.go:28] interesting pod/console-fcbb9b8d-nl248 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body=
Sep 08 15:13:59.340015 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.339905    2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-fcbb9b8d-nl248" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused"
Sep 08 15:13:59.753768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.753731    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qqhxl"]
Sep 08 15:13:59.757278 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.757260    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.760438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.760164    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 15:13:59.760438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.760217    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9k4qk\""
Sep 08 15:13:59.760438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.760235    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 15:13:59.760438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.760167    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 15:13:59.760943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.760924    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 15:13:59.854598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854610    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854641    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-sys\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854690    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjmf\" (UniqueName: \"kubernetes.io/projected/e92d8002-ba45-476d-b4e5-8f97870d3756-kube-api-access-jpjmf\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854716    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-metrics-client-ca\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854784    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-textfile\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854813    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-tls\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854847    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-root\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.854883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.854873    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-wtmp\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956213 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956181    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjmf\" (UniqueName: \"kubernetes.io/projected/e92d8002-ba45-476d-b4e5-8f97870d3756-kube-api-access-jpjmf\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956213 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956215    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-metrics-client-ca\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956259    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-textfile\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956288    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-tls\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-root\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956378    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-wtmp\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956406    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956428    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956458    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-sys\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956512    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-root\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956544    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-sys\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956574    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-wtmp\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956819 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956670    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-textfile\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.956956 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956938    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-metrics-client-ca\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.957012 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.956951    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.958792 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.958772    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-tls\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.958852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.958802    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e92d8002-ba45-476d-b4e5-8f97870d3756-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:13:59.963783 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:13:59.963761    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjmf\" (UniqueName: \"kubernetes.io/projected/e92d8002-ba45-476d-b4e5-8f97870d3756-kube-api-access-jpjmf\") pod \"node-exporter-qqhxl\" (UID: \"e92d8002-ba45-476d-b4e5-8f97870d3756\") " pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:14:00.068697 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:00.068619    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqhxl"
Sep 08 15:14:00.081677 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:14:00.081651    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92d8002_ba45_476d_b4e5_8f97870d3756.slice/crio-781a44d132da593eb86e9daea2677c8eb5b0ee336e8e136be2756e6dd13e60dc WatchSource:0}: Error finding container 781a44d132da593eb86e9daea2677c8eb5b0ee336e8e136be2756e6dd13e60dc: Status 404 returned error can't find the container with id 781a44d132da593eb86e9daea2677c8eb5b0ee336e8e136be2756e6dd13e60dc
Sep 08 15:14:00.083216 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:00.083194    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rc2px"
Sep 08 15:14:00.134985 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:00.134950    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqhxl" event={"ID":"e92d8002-ba45-476d-b4e5-8f97870d3756","Type":"ContainerStarted","Data":"781a44d132da593eb86e9daea2677c8eb5b0ee336e8e136be2756e6dd13e60dc"}
Sep 08 15:14:02.146796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:02.146753    2568 generic.go:356] "Generic (PLEG): container finished" podID="e92d8002-ba45-476d-b4e5-8f97870d3756" containerID="3d415ac93490d452874e2c014f415541dfbbb34741c8ae7ddca90d317ef95b91" exitCode=0
Sep 08 15:14:02.147227 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:02.146815    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqhxl" event={"ID":"e92d8002-ba45-476d-b4e5-8f97870d3756","Type":"ContainerDied","Data":"3d415ac93490d452874e2c014f415541dfbbb34741c8ae7ddca90d317ef95b91"}
Sep 08 15:14:03.154067 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:03.154032    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqhxl" event={"ID":"e92d8002-ba45-476d-b4e5-8f97870d3756","Type":"ContainerStarted","Data":"debe71eab1d6b9dc302263346085a6fd010737ea4e94c3698baddf71f34300ac"}
Sep 08 15:14:03.154577 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:03.154083    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqhxl" event={"ID":"e92d8002-ba45-476d-b4e5-8f97870d3756","Type":"ContainerStarted","Data":"ef5ca6097ed7f9aac3352399b356cfa64029bb08770cb77a8cfea1bf95819586"}
Sep 08 15:14:03.173223 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:03.173173    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qqhxl" podStartSLOduration=2.956614454 podStartE2EDuration="4.17315737s" totalImagesPullingTime="1.156721493s" totalInitContainerRuntime="59.821423ms" isStatefulPod=true podCreationTimestamp="2026-09-08 15:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:03.172925964 +0000 UTC m=+85.015723179" watchObservedRunningTime="2026-09-08 15:14:03.17315737 +0000 UTC m=+85.015954593"
Sep 08 15:14:04.211674 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:04.211643    2568 patch_prober.go:28] interesting pod/image-registry-76c7b9d8c9-r456z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 15:14:04.212028 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:04.211692    2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:14:04.460982 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:04.460943    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:14:06.048517 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:06.048490    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:14:09.341661 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:09.341626    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:14:09.345268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:09.345242    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:14:10.666736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:10.666698    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:14:15.316768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:15.316733    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:14:29.480819 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.480779    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bd97657dc-z7sx8" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerName="console" containerID="cri-o://d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17" gracePeriod=15
Sep 08 15:14:29.724768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.724746    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:14:29.795063 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.794990    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzvnz\" (UniqueName: \"kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795063 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795031    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795063 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795057    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795088    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795124    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795141    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config\") pod \"cef0bea7-3573-4183-b726-9fb4cadb3743\" (UID: \"cef0bea7-3573-4183-b726-9fb4cadb3743\") "
Sep 08 15:14:29.795879 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795530    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:29.795879 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795672    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config" (OuterVolumeSpecName: "console-config") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:29.795879 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.795838    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca" (OuterVolumeSpecName: "service-ca") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:29.797600 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.797566    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:29.797600 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.797581    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz" (OuterVolumeSpecName: "kube-api-access-tzvnz") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "kube-api-access-tzvnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:14:29.797730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.797610    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cef0bea7-3573-4183-b726-9fb4cadb3743" (UID: "cef0bea7-3573-4183-b726-9fb4cadb3743"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:29.896012 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.895987    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvnz\" (UniqueName: \"kubernetes.io/projected/cef0bea7-3573-4183-b726-9fb4cadb3743-kube-api-access-tzvnz\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:29.896012 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.896010    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:29.896142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.896022    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:29.896142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.896030    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-oauth-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:29.896142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.896039    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef0bea7-3573-4183-b726-9fb4cadb3743-console-oauth-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:29.896142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:29.896048    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef0bea7-3573-4183-b726-9fb4cadb3743-console-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:30.263050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.263020    2568 generic.go:356] "Generic (PLEG): container finished" podID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerID="d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17" exitCode=0
Sep 08 15:14:30.263215 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.263068    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd97657dc-z7sx8"
Sep 08 15:14:30.263215 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.263108    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd97657dc-z7sx8" event={"ID":"cef0bea7-3573-4183-b726-9fb4cadb3743","Type":"ContainerDied","Data":"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"}
Sep 08 15:14:30.263215 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.263158    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd97657dc-z7sx8" event={"ID":"cef0bea7-3573-4183-b726-9fb4cadb3743","Type":"ContainerDied","Data":"5f2359a53f72325abf03d18f780723e30c315def871e255d60117d999f89f4ef"}
Sep 08 15:14:30.263215 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.263181    2568 scope.go:117] "RemoveContainer" containerID="d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"
Sep 08 15:14:30.271915 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.271900    2568 scope.go:117] "RemoveContainer" containerID="d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"
Sep 08 15:14:30.272185 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:14:30.272165    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17\": container with ID starting with d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17 not found: ID does not exist" containerID="d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"
Sep 08 15:14:30.272256 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.272193    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17"} err="failed to get container status \"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17\": rpc error: code = NotFound desc = could not find container \"d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17\": container with ID starting with d064e95bb660f7395696bf1399a1b829142e893f74f13c9886712d2bf5a80b17 not found: ID does not exist"
Sep 08 15:14:30.283026 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.283004    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:14:30.287187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.287167    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd97657dc-z7sx8"]
Sep 08 15:14:30.674645 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:30.674611    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" path="/var/lib/kubelet/pods/cef0bea7-3573-4183-b726-9fb4cadb3743/volumes"
Sep 08 15:14:35.687606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:35.687558    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerName="registry" containerID="cri-o://ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886" gracePeriod=30
Sep 08 15:14:35.928841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:35.928815    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:14:36.044381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044284    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044360    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-676ml\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044379    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044401    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044428    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044464    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044503    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044548    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets\") pod \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\" (UID: \"19890a46-56dc-4ba3-af9a-0ce77e2c5f35\") "
Sep 08 15:14:36.044888 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.044853    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:36.045416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.045374    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:36.047122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.047082    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:36.047231 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.047204    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:14:36.047310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.047273    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:36.047375 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.047344    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml" (OuterVolumeSpecName: "kube-api-access-676ml") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "kube-api-access-676ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:14:36.047413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.047398    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:14:36.055560 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.055533    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "19890a46-56dc-4ba3-af9a-0ce77e2c5f35" (UID: "19890a46-56dc-4ba3-af9a-0ce77e2c5f35"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:14:36.145178 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145152    2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-image-registry-private-configuration\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145178 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145174    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676ml\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-kube-api-access-676ml\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145184    2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-certificates\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145194    2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-bound-sa-token\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145204    2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-registry-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145212    2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-ca-trust-extracted\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145220    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-trusted-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.145331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.145227    2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/19890a46-56dc-4ba3-af9a-0ce77e2c5f35-installation-pull-secrets\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:36.286955 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.286919    2568 generic.go:356] "Generic (PLEG): container finished" podID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerID="ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886" exitCode=0
Sep 08 15:14:36.287093 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.286969    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z"
Sep 08 15:14:36.287093 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.286980    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" event={"ID":"19890a46-56dc-4ba3-af9a-0ce77e2c5f35","Type":"ContainerDied","Data":"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"}
Sep 08 15:14:36.287093 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.287033    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76c7b9d8c9-r456z" event={"ID":"19890a46-56dc-4ba3-af9a-0ce77e2c5f35","Type":"ContainerDied","Data":"1f8854d0488696bd36de8ca1ea1ba3688172cb7bfe45e7726785fa63d04e5d69"}
Sep 08 15:14:36.287093 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.287059    2568 scope.go:117] "RemoveContainer" containerID="ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"
Sep 08 15:14:36.295359 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.295337    2568 scope.go:117] "RemoveContainer" containerID="ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"
Sep 08 15:14:36.295630 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:14:36.295612    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886\": container with ID starting with ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886 not found: ID does not exist" containerID="ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"
Sep 08 15:14:36.295694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.295636    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886"} err="failed to get container status \"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886\": rpc error: code = NotFound desc = could not find container \"ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886\": container with ID starting with ccd9a32bb66a8d8b18ff2b9914efd001e5eafa87da9de6b99140733f281af886 not found: ID does not exist"
Sep 08 15:14:36.307598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.307572    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:14:36.313092 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.313068    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76c7b9d8c9-r456z"]
Sep 08 15:14:36.678519 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:36.678488    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" path="/var/lib/kubelet/pods/19890a46-56dc-4ba3-af9a-0ce77e2c5f35/volumes"
Sep 08 15:14:40.343618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.343562    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fcbb9b8d-nl248" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerName="console" containerID="cri-o://8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768" gracePeriod=15
Sep 08 15:14:40.580841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.580822    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:14:40.677378 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677352    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677427    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqxvn\" (UniqueName: \"kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677446    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677477    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677607    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677656    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677740 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677684    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:40.677851 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677705    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle\") pod \"4588a581-7987-4d53-9641-8fbaec6c87e1\" (UID: \"4588a581-7987-4d53-9641-8fbaec6c87e1\") "
Sep 08 15:14:40.677943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.677890    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:40.678083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.678056    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:40.678198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.678077    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config" (OuterVolumeSpecName: "console-config") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:14:40.678198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.678132    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-trusted-ca-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.678198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.678153    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.678198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.678168    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-oauth-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.679771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.679744    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:40.679854 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.679773    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn" (OuterVolumeSpecName: "kube-api-access-sqxvn") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "kube-api-access-sqxvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:14:40.679854 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.679781    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4588a581-7987-4d53-9641-8fbaec6c87e1" (UID: "4588a581-7987-4d53-9641-8fbaec6c87e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:14:40.778691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.778666    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxvn\" (UniqueName: \"kubernetes.io/projected/4588a581-7987-4d53-9641-8fbaec6c87e1-kube-api-access-sqxvn\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.778691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.778687    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-oauth-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.778833 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.778696    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4588a581-7987-4d53-9641-8fbaec6c87e1-console-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:40.778833 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:40.778705    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4588a581-7987-4d53-9641-8fbaec6c87e1-console-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:14:41.307219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.307177    2568 generic.go:356] "Generic (PLEG): container finished" podID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerID="8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768" exitCode=0
Sep 08 15:14:41.307432 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.307222    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fcbb9b8d-nl248" event={"ID":"4588a581-7987-4d53-9641-8fbaec6c87e1","Type":"ContainerDied","Data":"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"}
Sep 08 15:14:41.307432 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.307240    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fcbb9b8d-nl248"
Sep 08 15:14:41.307432 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.307278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fcbb9b8d-nl248" event={"ID":"4588a581-7987-4d53-9641-8fbaec6c87e1","Type":"ContainerDied","Data":"4ecfa123bc2457c9a06d5a46ec242803a8d791abbea05a462085f5d6a3d7cac8"}
Sep 08 15:14:41.307432 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.307319    2568 scope.go:117] "RemoveContainer" containerID="8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"
Sep 08 15:14:41.315955 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.315938    2568 scope.go:117] "RemoveContainer" containerID="8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"
Sep 08 15:14:41.336392 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:14:41.336277    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768\": container with ID starting with 8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768 not found: ID does not exist" containerID="8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"
Sep 08 15:14:41.336392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.336344    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768"} err="failed to get container status \"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768\": rpc error: code = NotFound desc = could not find container \"8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768\": container with ID starting with 8cc21ffcee1186eaa6f01c252ad0ea9e337bb9e01374fbc914ba20c59ddd3768 not found: ID does not exist"
Sep 08 15:14:41.344530 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.344508    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:14:41.350401 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:41.350381    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fcbb9b8d-nl248"]
Sep 08 15:14:42.313868 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:42.313834    2568 generic.go:356] "Generic (PLEG): container finished" podID="712806b0-1710-4dda-8d6a-69e59e3c1839" containerID="496dd2ce4f37003585ade49c7c12ceb3a225a75192da9a1f70cbc23834d179c6" exitCode=0
Sep 08 15:14:42.314035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:42.313885    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs" event={"ID":"712806b0-1710-4dda-8d6a-69e59e3c1839","Type":"ContainerDied","Data":"496dd2ce4f37003585ade49c7c12ceb3a225a75192da9a1f70cbc23834d179c6"}
Sep 08 15:14:42.314168 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:42.314154    2568 scope.go:117] "RemoveContainer" containerID="496dd2ce4f37003585ade49c7c12ceb3a225a75192da9a1f70cbc23834d179c6"
Sep 08 15:14:42.674529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:42.674502    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" path="/var/lib/kubelet/pods/4588a581-7987-4d53-9641-8fbaec6c87e1/volumes"
Sep 08 15:14:43.319743 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:43.319713    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-n9hvs" event={"ID":"712806b0-1710-4dda-8d6a-69e59e3c1839","Type":"ContainerStarted","Data":"96aa51465b6c1da4c185216396f2bbd412bde77b7b4329645588c2b7d6fc6d4d"}
Sep 08 15:14:47.335906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:47.335827    2568 generic.go:356] "Generic (PLEG): container finished" podID="8aac3e49-399c-4f97-a4b0-9343e12bc985" containerID="c8e1e136507b3b7c13d8affa4a7004f77d6ca038162a1fd627251af153545d23" exitCode=0
Sep 08 15:14:47.335906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:47.335869    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx" event={"ID":"8aac3e49-399c-4f97-a4b0-9343e12bc985","Type":"ContainerDied","Data":"c8e1e136507b3b7c13d8affa4a7004f77d6ca038162a1fd627251af153545d23"}
Sep 08 15:14:47.336359 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:47.336144    2568 scope.go:117] "RemoveContainer" containerID="c8e1e136507b3b7c13d8affa4a7004f77d6ca038162a1fd627251af153545d23"
Sep 08 15:14:48.341688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:48.341657    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-fnjnx" event={"ID":"8aac3e49-399c-4f97-a4b0-9343e12bc985","Type":"ContainerStarted","Data":"a23bced81af27533e6c59ea3e032e0424cef0629c382082fdb9b2e1fecdd0d0f"}
Sep 08 15:14:52.361486 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:52.361444    2568 generic.go:356] "Generic (PLEG): container finished" podID="125e8c76-09e0-466a-8518-6469c0b019eb" containerID="e527e4f59a375b740a469f69cd4cb2d87704b3677159196df3ded3c61a07fd36" exitCode=0
Sep 08 15:14:52.361875 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:52.361518    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x" event={"ID":"125e8c76-09e0-466a-8518-6469c0b019eb","Type":"ContainerDied","Data":"e527e4f59a375b740a469f69cd4cb2d87704b3677159196df3ded3c61a07fd36"}
Sep 08 15:14:52.361950 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:52.361927    2568 scope.go:117] "RemoveContainer" containerID="e527e4f59a375b740a469f69cd4cb2d87704b3677159196df3ded3c61a07fd36"
Sep 08 15:14:53.366723 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:14:53.366689    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fdg9x" event={"ID":"125e8c76-09e0-466a-8518-6469c0b019eb","Type":"ContainerStarted","Data":"602a3c858cc9c399e330f0fa7134d66d7ce12bfc459b0549597ad245f5ebbae4"}
Sep 08 15:15:27.649486 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649453    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649861    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerName="console"
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649881    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerName="console"
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649912    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerName="console"
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649921    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerName="console"
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649939    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerName="registry"
Sep 08 15:15:27.649989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.649948    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerName="registry"
Sep 08 15:15:27.650268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.650017    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef0bea7-3573-4183-b726-9fb4cadb3743" containerName="console"
Sep 08 15:15:27.650268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.650029    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="19890a46-56dc-4ba3-af9a-0ce77e2c5f35" containerName="registry"
Sep 08 15:15:27.650268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.650040    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4588a581-7987-4d53-9641-8fbaec6c87e1" containerName="console"
Sep 08 15:15:27.659567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.659546    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.661900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.661875    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 15:15:27.662758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662732    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p8lf5\""
Sep 08 15:15:27.663085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662890    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 15:15:27.663085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662946    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 15:15:27.663085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662967    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 15:15:27.663085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662846    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 15:15:27.663085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.662844    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 15:15:27.663448 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.663252    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 15:15:27.665933 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.665905    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:15:27.669599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.669578    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 15:15:27.722731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.722696    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.722731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.722732    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.722917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.722760    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.722917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.722817    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.722997 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.722949    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7pk\" (UniqueName: \"kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.723047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.723011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.723140 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.723124    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823536    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823568    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7pk\" (UniqueName: \"kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823609    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823637    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823683    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.823728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.823712    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.824356 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.824330    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.824506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.824480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.824506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.824492    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.824667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.824418    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.826058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.826038    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.826123 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.826058    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.831419 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.831397    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7pk\" (UniqueName: \"kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk\") pod \"console-dcbd47969-cl566\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") " pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:27.972515 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:27.972435    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:28.244671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:28.244601    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:15:28.249677 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:15:28.249650    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8db5082_e166_49da_b714_50c5c9458aa4.slice/crio-bd01a2f7203ccb0b5848c3ef36d0a8df7fcf3c80998d6a6f78027100a558e894 WatchSource:0}: Error finding container bd01a2f7203ccb0b5848c3ef36d0a8df7fcf3c80998d6a6f78027100a558e894: Status 404 returned error can't find the container with id bd01a2f7203ccb0b5848c3ef36d0a8df7fcf3c80998d6a6f78027100a558e894
Sep 08 15:15:28.508227 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:28.508136    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd47969-cl566" event={"ID":"d8db5082-e166-49da-b714-50c5c9458aa4","Type":"ContainerStarted","Data":"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"}
Sep 08 15:15:28.508227 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:28.508199    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd47969-cl566" event={"ID":"d8db5082-e166-49da-b714-50c5c9458aa4","Type":"ContainerStarted","Data":"bd01a2f7203ccb0b5848c3ef36d0a8df7fcf3c80998d6a6f78027100a558e894"}
Sep 08 15:15:28.527427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:28.527376    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcbd47969-cl566" podStartSLOduration=1.527361102 podStartE2EDuration="1.527361102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:15:28.525847829 +0000 UTC m=+170.368645053" watchObservedRunningTime="2026-09-08 15:15:28.527361102 +0000 UTC m=+170.370158324"
Sep 08 15:15:37.973440 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:37.973353    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:37.973440 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:37.973416    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:37.978321 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:37.978286    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:38.548987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.548956    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:15:38.728664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.728624    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:15:38.733044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.733015    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.743719 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.743694    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:15:38.807265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807191    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807224    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478l7\" (UniqueName: \"kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807336    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807369    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807385    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.807514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.807402    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908246 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908211    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908266    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478l7\" (UniqueName: \"kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908336    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908376    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908400    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.908479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.908427    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.909053 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.909026    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.909206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.909178    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.909329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.909072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.909329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.909072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.910817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.910794    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.911027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.911006    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:38.916163 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:38.916145    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-478l7\" (UniqueName: \"kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7\") pod \"console-7f77845f55-bh5nq\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") " pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:39.043986 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:39.043950    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:39.319243 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:39.319216    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:15:39.321948 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:15:39.321919    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58347eb5_0ba1_4d63_9029_47ba6f4372a0.slice/crio-16223fba04a485c4cb89b6719ff663b50ba7c40960e7dd1be39fc658ba29bc6d WatchSource:0}: Error finding container 16223fba04a485c4cb89b6719ff663b50ba7c40960e7dd1be39fc658ba29bc6d: Status 404 returned error can't find the container with id 16223fba04a485c4cb89b6719ff663b50ba7c40960e7dd1be39fc658ba29bc6d
Sep 08 15:15:39.550608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:39.550571    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f77845f55-bh5nq" event={"ID":"58347eb5-0ba1-4d63-9029-47ba6f4372a0","Type":"ContainerStarted","Data":"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"}
Sep 08 15:15:39.551005 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:39.550625    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f77845f55-bh5nq" event={"ID":"58347eb5-0ba1-4d63-9029-47ba6f4372a0","Type":"ContainerStarted","Data":"16223fba04a485c4cb89b6719ff663b50ba7c40960e7dd1be39fc658ba29bc6d"}
Sep 08 15:15:39.573169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:39.573104    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f77845f55-bh5nq" podStartSLOduration=1.5730831109999999 podStartE2EDuration="1.573083111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:15:39.566121147 +0000 UTC m=+181.408918365" watchObservedRunningTime="2026-09-08 15:15:39.573083111 +0000 UTC m=+181.415880334"
Sep 08 15:15:49.044677 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:49.044642    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:49.045159 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:49.044710    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:49.049397 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:49.049374    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:49.592019 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:49.591992    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:15:49.634497 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:15:49.634469    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:16:14.656748 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.656688    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dcbd47969-cl566" podUID="d8db5082-e166-49da-b714-50c5c9458aa4" containerName="console" containerID="cri-o://6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3" gracePeriod=15
Sep 08 15:16:14.893345 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.893322    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:16:14.994491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994412    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th7pk\" (UniqueName: \"kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994454    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994475    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994504    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994544    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994590    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.994736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994627    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config\") pod \"d8db5082-e166-49da-b714-50c5c9458aa4\" (UID: \"d8db5082-e166-49da-b714-50c5c9458aa4\") "
Sep 08 15:16:14.995018 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.994986    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:16:14.995143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.995012    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:16:14.995143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.995016    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config" (OuterVolumeSpecName: "console-config") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:16:14.995143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.995026    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:16:14.996735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.996706    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:16:14.996821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.996746    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk" (OuterVolumeSpecName: "kube-api-access-th7pk") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "kube-api-access-th7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:16:14.996821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:14.996759    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8db5082-e166-49da-b714-50c5c9458aa4" (UID: "d8db5082-e166-49da-b714-50c5c9458aa4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:16:15.096179 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096145    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-oauth-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096179 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096171    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th7pk\" (UniqueName: \"kubernetes.io/projected/d8db5082-e166-49da-b714-50c5c9458aa4-kube-api-access-th7pk\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096179 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096180    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-trusted-ca-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096190    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-console-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096198    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096206    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8db5082-e166-49da-b714-50c5c9458aa4-oauth-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.096426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.096214    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8db5082-e166-49da-b714-50c5c9458aa4-console-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:15.687983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.687951    2568 generic.go:356] "Generic (PLEG): container finished" podID="d8db5082-e166-49da-b714-50c5c9458aa4" containerID="6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3" exitCode=0
Sep 08 15:16:15.688436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.687990    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcbd47969-cl566"
Sep 08 15:16:15.688436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.687995    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd47969-cl566" event={"ID":"d8db5082-e166-49da-b714-50c5c9458aa4","Type":"ContainerDied","Data":"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"}
Sep 08 15:16:15.688436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.688118    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcbd47969-cl566" event={"ID":"d8db5082-e166-49da-b714-50c5c9458aa4","Type":"ContainerDied","Data":"bd01a2f7203ccb0b5848c3ef36d0a8df7fcf3c80998d6a6f78027100a558e894"}
Sep 08 15:16:15.688436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.688145    2568 scope.go:117] "RemoveContainer" containerID="6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"
Sep 08 15:16:15.699148 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.699133    2568 scope.go:117] "RemoveContainer" containerID="6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"
Sep 08 15:16:15.699417 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:16:15.699398    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3\": container with ID starting with 6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3 not found: ID does not exist" containerID="6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"
Sep 08 15:16:15.699482 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.699426    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3"} err="failed to get container status \"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3\": rpc error: code = NotFound desc = could not find container \"6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3\": container with ID starting with 6b122468c1a2b044e73aaab22ea883fae1222a94d523b60b0bd30b6f5be6a6f3 not found: ID does not exist"
Sep 08 15:16:15.709042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.709022    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:16:15.713445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:15.713425    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcbd47969-cl566"]
Sep 08 15:16:16.675237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:16.675204    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8db5082-e166-49da-b714-50c5c9458aa4" path="/var/lib/kubelet/pods/d8db5082-e166-49da-b714-50c5c9458aa4/volumes"
Sep 08 15:16:29.739883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.739845    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"]
Sep 08 15:16:29.740412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.740127    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8db5082-e166-49da-b714-50c5c9458aa4" containerName="console"
Sep 08 15:16:29.740412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.740138    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8db5082-e166-49da-b714-50c5c9458aa4" containerName="console"
Sep 08 15:16:29.740412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.740198    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8db5082-e166-49da-b714-50c5c9458aa4" containerName="console"
Sep 08 15:16:29.744462 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.744444    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.746698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.746667    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:16:29.746815 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.746735    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:16:29.746815 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.746744    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rprqj\""
Sep 08 15:16:29.751770 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.751749    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"]
Sep 08 15:16:29.803051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.803022    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxg7\" (UniqueName: \"kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.803189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.803059    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.803189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.803133    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.904490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.904456    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxg7\" (UniqueName: \"kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.904626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.904500    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.904626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.904557    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.904906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.904884    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.904950 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.904932    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:29.912425 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:29.912403    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxg7\" (UniqueName: \"kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:30.054942 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:30.054859    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:30.323785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:30.323705    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"]
Sep 08 15:16:30.327093 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:16:30.327061    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d5a4ec_3518_4398_8779_884241d50319.slice/crio-8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc WatchSource:0}: Error finding container 8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc: Status 404 returned error can't find the container with id 8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc
Sep 08 15:16:30.755329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:30.755278    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q" event={"ID":"d9d5a4ec-3518-4398-8779-884241d50319","Type":"ContainerStarted","Data":"8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc"}
Sep 08 15:16:39.791813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:39.791773    2568 generic.go:356] "Generic (PLEG): container finished" podID="d9d5a4ec-3518-4398-8779-884241d50319" containerID="0fcf6dd0138dad823446cb04ebf4e3d7cb3f8a318e70a91e47af143a2b8ca66b" exitCode=0
Sep 08 15:16:39.792174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:39.791851    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q" event={"ID":"d9d5a4ec-3518-4398-8779-884241d50319","Type":"ContainerDied","Data":"0fcf6dd0138dad823446cb04ebf4e3d7cb3f8a318e70a91e47af143a2b8ca66b"}
Sep 08 15:16:42.806088 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:42.806055    2568 generic.go:356] "Generic (PLEG): container finished" podID="d9d5a4ec-3518-4398-8779-884241d50319" containerID="8561bc49e9bdb3790bb005528d15d2941399e7dff339a0ef92b8e1a8569eeb41" exitCode=0
Sep 08 15:16:42.806467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:42.806141    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q" event={"ID":"d9d5a4ec-3518-4398-8779-884241d50319","Type":"ContainerDied","Data":"8561bc49e9bdb3790bb005528d15d2941399e7dff339a0ef92b8e1a8569eeb41"}
Sep 08 15:16:50.841766 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:50.841732    2568 generic.go:356] "Generic (PLEG): container finished" podID="d9d5a4ec-3518-4398-8779-884241d50319" containerID="7c53d495fcf7ee133ecbb2462036de9b354ab7b9313f95dc32f599a2f1da08f1" exitCode=0
Sep 08 15:16:50.842157 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:50.841818    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q" event={"ID":"d9d5a4ec-3518-4398-8779-884241d50319","Type":"ContainerDied","Data":"7c53d495fcf7ee133ecbb2462036de9b354ab7b9313f95dc32f599a2f1da08f1"}
Sep 08 15:16:52.017061 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.017034    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:52.087646 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.087617    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmxg7\" (UniqueName: \"kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7\") pod \"d9d5a4ec-3518-4398-8779-884241d50319\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") "
Sep 08 15:16:52.087782 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.087662    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle\") pod \"d9d5a4ec-3518-4398-8779-884241d50319\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") "
Sep 08 15:16:52.087782 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.087714    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util\") pod \"d9d5a4ec-3518-4398-8779-884241d50319\" (UID: \"d9d5a4ec-3518-4398-8779-884241d50319\") "
Sep 08 15:16:52.088371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.088347    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle" (OuterVolumeSpecName: "bundle") pod "d9d5a4ec-3518-4398-8779-884241d50319" (UID: "d9d5a4ec-3518-4398-8779-884241d50319"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:16:52.089791 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.089772    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7" (OuterVolumeSpecName: "kube-api-access-qmxg7") pod "d9d5a4ec-3518-4398-8779-884241d50319" (UID: "d9d5a4ec-3518-4398-8779-884241d50319"). InnerVolumeSpecName "kube-api-access-qmxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:16:52.092142 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.092119    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util" (OuterVolumeSpecName: "util") pod "d9d5a4ec-3518-4398-8779-884241d50319" (UID: "d9d5a4ec-3518-4398-8779-884241d50319"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:16:52.188237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.188209    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:52.188237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.188235    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9d5a4ec-3518-4398-8779-884241d50319-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:52.188395 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.188245    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmxg7\" (UniqueName: \"kubernetes.io/projected/d9d5a4ec-3518-4398-8779-884241d50319-kube-api-access-qmxg7\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:16:52.851027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.850997    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q"
Sep 08 15:16:52.851188 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.850997    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540b6m7q" event={"ID":"d9d5a4ec-3518-4398-8779-884241d50319","Type":"ContainerDied","Data":"8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc"}
Sep 08 15:16:52.851188 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:52.851125    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8198d8075bb11283faff4833554e22e3a4eb7b6d679813d4590c39ac4f899ccc"
Sep 08 15:16:57.769490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769455    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"]
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769722    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="pull"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769733    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="pull"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769753    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="util"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769759    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="util"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769765    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="extract"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769771    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="extract"
Sep 08 15:16:57.769852 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.769818    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d5a4ec-3518-4398-8779-884241d50319" containerName="extract"
Sep 08 15:16:57.772461 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.772444    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:57.774835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.774800    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 08 15:16:57.774835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.774822    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:16:57.775000 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.774906    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lbl9n\""
Sep 08 15:16:57.784126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.784102    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"]
Sep 08 15:16:57.932234 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.932200    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vwn\" (UniqueName: \"kubernetes.io/projected/f126988b-0aa8-4bc5-83b8-31934739c2e8-kube-api-access-79vwn\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:57.932234 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:57.932237    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f126988b-0aa8-4bc5-83b8-31934739c2e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.032972 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.032893    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vwn\" (UniqueName: \"kubernetes.io/projected/f126988b-0aa8-4bc5-83b8-31934739c2e8-kube-api-access-79vwn\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.032972 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.032930    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f126988b-0aa8-4bc5-83b8-31934739c2e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.033410 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.033387    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f126988b-0aa8-4bc5-83b8-31934739c2e8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.041781 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.041759    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vwn\" (UniqueName: \"kubernetes.io/projected/f126988b-0aa8-4bc5-83b8-31934739c2e8-kube-api-access-79vwn\") pod \"cert-manager-operator-controller-manager-55c444bf86-r8q68\" (UID: \"f126988b-0aa8-4bc5-83b8-31934739c2e8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.081229 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.081193    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"
Sep 08 15:16:58.355527 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.355448    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68"]
Sep 08 15:16:58.359092 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:16:58.359064    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf126988b_0aa8_4bc5_83b8_31934739c2e8.slice/crio-81de426ba4a59a4031087dd03540ffecf2fbf875fd831622fa2983718a3b7458 WatchSource:0}: Error finding container 81de426ba4a59a4031087dd03540ffecf2fbf875fd831622fa2983718a3b7458: Status 404 returned error can't find the container with id 81de426ba4a59a4031087dd03540ffecf2fbf875fd831622fa2983718a3b7458
Sep 08 15:16:58.875494 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:16:58.875458    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68" event={"ID":"f126988b-0aa8-4bc5-83b8-31934739c2e8","Type":"ContainerStarted","Data":"81de426ba4a59a4031087dd03540ffecf2fbf875fd831622fa2983718a3b7458"}
Sep 08 15:17:00.884886 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:00.884849    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68" event={"ID":"f126988b-0aa8-4bc5-83b8-31934739c2e8","Type":"ContainerStarted","Data":"5f070a6ac8dac0594d73852832a7802d8e712ad4afed8065c0d7953d326627c8"}
Sep 08 15:17:00.904214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:00.904170    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-r8q68" podStartSLOduration=2.397941201 podStartE2EDuration="3.904157093s" totalImagesPullingTime="1.506215892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:00.903016297 +0000 UTC m=+262.745813520" watchObservedRunningTime="2026-09-08 15:17:00.904157093 +0000 UTC m=+262.746954316"
Sep 08 15:17:01.559700 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.559665    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"]
Sep 08 15:17:01.563038 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.563021    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.569822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.569803    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:17:01.569894 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.569846    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rprqj\""
Sep 08 15:17:01.569950 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.569912    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:17:01.585530 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.585507    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"]
Sep 08 15:17:01.659828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.659798    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.659828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.659830    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzgl\" (UniqueName: \"kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.660019 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.659864    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.760946 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.760912    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.761127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.760979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.761127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.760998    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzgl\" (UniqueName: \"kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.761361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.761330    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.761474 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.761380    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.778090 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.778060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzgl\" (UniqueName: \"kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:01.872301 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:01.872259    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:02.142364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:02.142275    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"]
Sep 08 15:17:02.145973 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:02.145939    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8925f7c2_8972_4604_8286_240f19e7de35.slice/crio-3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f WatchSource:0}: Error finding container 3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f: Status 404 returned error can't find the container with id 3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f
Sep 08 15:17:02.894843 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:02.894806    2568 generic.go:356] "Generic (PLEG): container finished" podID="8925f7c2-8972-4604-8286-240f19e7de35" containerID="a04bed5973ff491bdc3260e522740891828a6b22ca4bb534c5c3cafdee14480e" exitCode=0
Sep 08 15:17:02.895027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:02.894862    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh" event={"ID":"8925f7c2-8972-4604-8286-240f19e7de35","Type":"ContainerDied","Data":"a04bed5973ff491bdc3260e522740891828a6b22ca4bb534c5c3cafdee14480e"}
Sep 08 15:17:02.895027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:02.894909    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh" event={"ID":"8925f7c2-8972-4604-8286-240f19e7de35","Type":"ContainerStarted","Data":"3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f"}
Sep 08 15:17:05.915263 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:05.915228    2568 generic.go:356] "Generic (PLEG): container finished" podID="8925f7c2-8972-4604-8286-240f19e7de35" containerID="7b8961f8b838089d5d3098a860436077b43a68451fcea8e5ab98953a6b4bffc7" exitCode=0
Sep 08 15:17:05.915654 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:05.915272    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh" event={"ID":"8925f7c2-8972-4604-8286-240f19e7de35","Type":"ContainerDied","Data":"7b8961f8b838089d5d3098a860436077b43a68451fcea8e5ab98953a6b4bffc7"}
Sep 08 15:17:06.922145 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:06.922110    2568 generic.go:356] "Generic (PLEG): container finished" podID="8925f7c2-8972-4604-8286-240f19e7de35" containerID="0d1f65463bc867ab59b1a35754aea4b230930e86a0f16ffd4c664ee6276d79fd" exitCode=0
Sep 08 15:17:06.922538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:06.922156    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh" event={"ID":"8925f7c2-8972-4604-8286-240f19e7de35","Type":"ContainerDied","Data":"0d1f65463bc867ab59b1a35754aea4b230930e86a0f16ffd4c664ee6276d79fd"}
Sep 08 15:17:08.046460 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.046437    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:08.211157 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.211074    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvzgl\" (UniqueName: \"kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl\") pod \"8925f7c2-8972-4604-8286-240f19e7de35\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") "
Sep 08 15:17:08.211157 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.211118    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util\") pod \"8925f7c2-8972-4604-8286-240f19e7de35\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") "
Sep 08 15:17:08.211157 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.211159    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle\") pod \"8925f7c2-8972-4604-8286-240f19e7de35\" (UID: \"8925f7c2-8972-4604-8286-240f19e7de35\") "
Sep 08 15:17:08.211580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.211555    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle" (OuterVolumeSpecName: "bundle") pod "8925f7c2-8972-4604-8286-240f19e7de35" (UID: "8925f7c2-8972-4604-8286-240f19e7de35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:08.213131 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.213110    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl" (OuterVolumeSpecName: "kube-api-access-wvzgl") pod "8925f7c2-8972-4604-8286-240f19e7de35" (UID: "8925f7c2-8972-4604-8286-240f19e7de35"). InnerVolumeSpecName "kube-api-access-wvzgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:17:08.215642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.215603    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util" (OuterVolumeSpecName: "util") pod "8925f7c2-8972-4604-8286-240f19e7de35" (UID: "8925f7c2-8972-4604-8286-240f19e7de35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:08.312540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.312508    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzgl\" (UniqueName: \"kubernetes.io/projected/8925f7c2-8972-4604-8286-240f19e7de35-kube-api-access-wvzgl\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:08.312540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.312534    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:08.312540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.312543    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8925f7c2-8972-4604-8286-240f19e7de35-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:08.935121 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.935041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh" event={"ID":"8925f7c2-8972-4604-8286-240f19e7de35","Type":"ContainerDied","Data":"3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f"}
Sep 08 15:17:08.935121 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.935089    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3038a0d483d03811aeb241b6362eaab1dbc039d2787a9f6c85b4f9b884b1085f"
Sep 08 15:17:08.935121 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:08.935045    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8w4xh"
Sep 08 15:17:13.070200 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070163    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t4m9h"]
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070461    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="extract"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070472    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="extract"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070484    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="util"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070489    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="util"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070496    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="pull"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070502    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="pull"
Sep 08 15:17:13.070635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.070552    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8925f7c2-8972-4604-8286-240f19e7de35" containerName="extract"
Sep 08 15:17:13.074505 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.074489    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.076679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.076655    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 15:17:13.076798 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.076754    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 15:17:13.077431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.077409    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2wt9g\""
Sep 08 15:17:13.083020 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.083000    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t4m9h"]
Sep 08 15:17:13.249364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.249332    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2sz\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-kube-api-access-kn2sz\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.249364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.249366    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-bound-sa-token\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.350143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.350070    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2sz\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-kube-api-access-kn2sz\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.350143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.350108    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-bound-sa-token\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.364802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.364776    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-bound-sa-token\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.368428 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.368404    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2sz\" (UniqueName: \"kubernetes.io/projected/f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0-kube-api-access-kn2sz\") pod \"cert-manager-55b4d954d5-t4m9h\" (UID: \"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0\") " pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.397274 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.397246    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-t4m9h"
Sep 08 15:17:13.426114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.426082    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-btml5"]
Sep 08 15:17:13.430058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.430033    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.432176 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.432154    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fcbvh\""
Sep 08 15:17:13.439486 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.439464    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-btml5"]
Sep 08 15:17:13.551343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.551313    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.551499 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.551365    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdlg\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-kube-api-access-tbdlg\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.652065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.652030    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.652225 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.652092    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdlg\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-kube-api-access-tbdlg\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.660482 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.660452    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.661416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.661393    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdlg\" (UniqueName: \"kubernetes.io/projected/1b46e3dc-7da7-4332-85e6-5f44b4185911-kube-api-access-tbdlg\") pod \"cert-manager-cainjector-559cdd8c87-btml5\" (UID: \"1b46e3dc-7da7-4332-85e6-5f44b4185911\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.669148 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.669127    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-t4m9h"]
Sep 08 15:17:13.671598 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:13.671578    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3a0cf3f_cd3a_4aab_9e6e_517a2b593fc0.slice/crio-9e356b615f5534689d84dde2f1d126c7d48b8ff40811c682f902688d31ea90a1 WatchSource:0}: Error finding container 9e356b615f5534689d84dde2f1d126c7d48b8ff40811c682f902688d31ea90a1: Status 404 returned error can't find the container with id 9e356b615f5534689d84dde2f1d126c7d48b8ff40811c682f902688d31ea90a1
Sep 08 15:17:13.741480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.741455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5"
Sep 08 15:17:13.958414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:13.958331    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-t4m9h" event={"ID":"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0","Type":"ContainerStarted","Data":"9e356b615f5534689d84dde2f1d126c7d48b8ff40811c682f902688d31ea90a1"}
Sep 08 15:17:14.008493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:14.008462    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-btml5"]
Sep 08 15:17:14.011172 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:14.011146    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b46e3dc_7da7_4332_85e6_5f44b4185911.slice/crio-6b1a9a422ace1076df88c2a4961705ef413d5a9f1a4f721bdb5a3a943c0b76bb WatchSource:0}: Error finding container 6b1a9a422ace1076df88c2a4961705ef413d5a9f1a4f721bdb5a3a943c0b76bb: Status 404 returned error can't find the container with id 6b1a9a422ace1076df88c2a4961705ef413d5a9f1a4f721bdb5a3a943c0b76bb
Sep 08 15:17:14.965765 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:14.965706    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5" event={"ID":"1b46e3dc-7da7-4332-85e6-5f44b4185911","Type":"ContainerStarted","Data":"6b1a9a422ace1076df88c2a4961705ef413d5a9f1a4f721bdb5a3a943c0b76bb"}
Sep 08 15:17:16.977648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:16.977600    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5" event={"ID":"1b46e3dc-7da7-4332-85e6-5f44b4185911","Type":"ContainerStarted","Data":"2ab05be34d0902810db806ef7ba57075a5a826418611f58ae12ee40480292ef8"}
Sep 08 15:17:16.979173 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:16.979146    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-t4m9h" event={"ID":"f3a0cf3f-cd3a-4aab-9e6e-517a2b593fc0","Type":"ContainerStarted","Data":"5982d7bb3d5a9926b5342e67b265ae38d3eb5015f12e053ef403559e4cf2b1e6"}
Sep 08 15:17:16.992563 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:16.992514    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-btml5" podStartSLOduration=1.219252475 podStartE2EDuration="3.992499693s" totalImagesPullingTime="2.773247218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:16.991948997 +0000 UTC m=+278.834746221" watchObservedRunningTime="2026-09-08 15:17:16.992499693 +0000 UTC m=+278.835296916"
Sep 08 15:17:17.006857 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:17.006796    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-t4m9h" podStartSLOduration=0.900492449 podStartE2EDuration="4.006772431s" totalImagesPullingTime="3.106279982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:17.006251075 +0000 UTC m=+278.849048301" watchObservedRunningTime="2026-09-08 15:17:17.006772431 +0000 UTC m=+278.849569655"
Sep 08 15:17:21.524445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.524413    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"]
Sep 08 15:17:21.527024 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.527007    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.529371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.529343    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:17:21.529500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.529349    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:17:21.529500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.529372    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rprqj\""
Sep 08 15:17:21.536126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.536103    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"]
Sep 08 15:17:21.616725 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.616699    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.616843 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.616735    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.616843 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.616765    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46d5s\" (UniqueName: \"kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.717130 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.717099    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.717130 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.717131    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.717336 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.717161    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46d5s\" (UniqueName: \"kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.717536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.717516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.717576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.717535    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.725049 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.725025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46d5s\" (UniqueName: \"kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:21.837133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:21.837072    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:22.107280 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:22.107255    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"]
Sep 08 15:17:22.109968 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:22.109934    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92100eb7_6e42_443c_8da4_889e8ef97ecf.slice/crio-5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e WatchSource:0}: Error finding container 5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e: Status 404 returned error can't find the container with id 5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e
Sep 08 15:17:23.005763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:23.005671    2568 generic.go:356] "Generic (PLEG): container finished" podID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerID="c492b9ee3a9b4a4496652fa31baf63ed1e9812e2c2d555ead2be463319dd904f" exitCode=0
Sep 08 15:17:23.006167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:23.005759    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz" event={"ID":"92100eb7-6e42-443c-8da4-889e8ef97ecf","Type":"ContainerDied","Data":"c492b9ee3a9b4a4496652fa31baf63ed1e9812e2c2d555ead2be463319dd904f"}
Sep 08 15:17:23.006167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:23.005809    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz" event={"ID":"92100eb7-6e42-443c-8da4-889e8ef97ecf","Type":"ContainerStarted","Data":"5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e"}
Sep 08 15:17:24.011834 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:24.011752    2568 generic.go:356] "Generic (PLEG): container finished" podID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerID="e3d14a7889bbf6296f7cf5a40c79db43cdc5af83c986bf06e3b936792946fb24" exitCode=0
Sep 08 15:17:24.012167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:24.011838    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz" event={"ID":"92100eb7-6e42-443c-8da4-889e8ef97ecf","Type":"ContainerDied","Data":"e3d14a7889bbf6296f7cf5a40c79db43cdc5af83c986bf06e3b936792946fb24"}
Sep 08 15:17:25.019682 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:25.019638    2568 generic.go:356] "Generic (PLEG): container finished" podID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerID="46bcc5a7259a26f4c264f28e5c5bae3969d66dd145b41296161177d6d8a6bc35" exitCode=0
Sep 08 15:17:25.020138 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:25.019686    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz" event={"ID":"92100eb7-6e42-443c-8da4-889e8ef97ecf","Type":"ContainerDied","Data":"46bcc5a7259a26f4c264f28e5c5bae3969d66dd145b41296161177d6d8a6bc35"}
Sep 08 15:17:26.149382 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.149358    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:26.246327 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.246278    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle\") pod \"92100eb7-6e42-443c-8da4-889e8ef97ecf\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") "
Sep 08 15:17:26.246512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.246335    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46d5s\" (UniqueName: \"kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s\") pod \"92100eb7-6e42-443c-8da4-889e8ef97ecf\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") "
Sep 08 15:17:26.246512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.246360    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util\") pod \"92100eb7-6e42-443c-8da4-889e8ef97ecf\" (UID: \"92100eb7-6e42-443c-8da4-889e8ef97ecf\") "
Sep 08 15:17:26.247038 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.247011    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle" (OuterVolumeSpecName: "bundle") pod "92100eb7-6e42-443c-8da4-889e8ef97ecf" (UID: "92100eb7-6e42-443c-8da4-889e8ef97ecf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:26.248469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.248440    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s" (OuterVolumeSpecName: "kube-api-access-46d5s") pod "92100eb7-6e42-443c-8da4-889e8ef97ecf" (UID: "92100eb7-6e42-443c-8da4-889e8ef97ecf"). InnerVolumeSpecName "kube-api-access-46d5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:17:26.252552 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.252517    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util" (OuterVolumeSpecName: "util") pod "92100eb7-6e42-443c-8da4-889e8ef97ecf" (UID: "92100eb7-6e42-443c-8da4-889e8ef97ecf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:26.347354 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.347262    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:26.347354 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.347312    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46d5s\" (UniqueName: \"kubernetes.io/projected/92100eb7-6e42-443c-8da4-889e8ef97ecf-kube-api-access-46d5s\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:26.347354 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:26.347326    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92100eb7-6e42-443c-8da4-889e8ef97ecf-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:27.030884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:27.030800    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz" event={"ID":"92100eb7-6e42-443c-8da4-889e8ef97ecf","Type":"ContainerDied","Data":"5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e"}
Sep 08 15:17:27.030884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:27.030844    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d7a2a52f6be675ca9fe77ff5437a718321af58a7b2e5d9ca9810afd3715f23e"
Sep 08 15:17:27.030884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:27.030842    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6wzz"
Sep 08 15:17:38.523412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523374    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"]
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523700    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="util"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523714    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="util"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523723    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="extract"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523728    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="extract"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523741    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="pull"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523747    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="pull"
Sep 08 15:17:38.523803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.523803    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="92100eb7-6e42-443c-8da4-889e8ef97ecf" containerName="extract"
Sep 08 15:17:38.526839 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.526823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.529559 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.529540    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rprqj\""
Sep 08 15:17:38.529642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.529541    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:17:38.529642 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.529543    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:17:38.539457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.539435    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"]
Sep 08 15:17:38.564427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.564403    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:17:38.567960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.567939    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:17:38.577022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.577003    2568 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 15:17:38.640308 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.640266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhkw\" (UniqueName: \"kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.642404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.640314    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.642404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.640385    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.740855 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.740825    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.741013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.740942    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhkw\" (UniqueName: \"kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.741013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.740972    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.741169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.741151    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.741273 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.741248    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.754508 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.754486    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhkw\" (UniqueName: \"kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:38.836216 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:38.836149    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:39.000324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.000282    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"]
Sep 08 15:17:39.003917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.003896    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.006478 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.006452    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 15:17:39.006618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.006495    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 15:17:39.006618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.006555    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 15:17:39.006618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.006581    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 15:17:39.006821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.006750    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xw2jq\""
Sep 08 15:17:39.022820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.022793    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"]
Sep 08 15:17:39.043357 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.043330    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mbt\" (UniqueName: \"kubernetes.io/projected/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-kube-api-access-x2mbt\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.043491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.043376    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-manifests\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.043491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.043393    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-tmp\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.043491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.043484    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.043598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.043506    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.123669 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.123641    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"]
Sep 08 15:17:39.126441 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:39.126417    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282a660b_1fad_49c7_a37c_c16266440f70.slice/crio-d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa WatchSource:0}: Error finding container d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa: Status 404 returned error can't find the container with id d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa
Sep 08 15:17:39.144344 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144316    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mbt\" (UniqueName: \"kubernetes.io/projected/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-kube-api-access-x2mbt\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.144449 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144368    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-manifests\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.144449 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-tmp\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.144561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144460    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.144561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144486    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.144801 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.144758    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-manifests\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.146798 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.146776    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-tmp\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.146906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.146828    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.147217 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.147197    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.166753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.166733    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mbt\" (UniqueName: \"kubernetes.io/projected/a7d5fc44-22a8-41fd-9178-6429f54dd5f8-kube-api-access-x2mbt\") pod \"opendatahub-operator-controller-manager-68dbb6d945-qn7m8\" (UID: \"a7d5fc44-22a8-41fd-9178-6429f54dd5f8\") " pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.314065 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.314027    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:39.594398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:39.594366    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"]
Sep 08 15:17:39.596418 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:17:39.596392    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d5fc44_22a8_41fd_9178_6429f54dd5f8.slice/crio-44fce72541cd477912c24e78e063a1b90aa93148ec9ec3d43ceefc14c22d2baa WatchSource:0}: Error finding container 44fce72541cd477912c24e78e063a1b90aa93148ec9ec3d43ceefc14c22d2baa: Status 404 returned error can't find the container with id 44fce72541cd477912c24e78e063a1b90aa93148ec9ec3d43ceefc14c22d2baa
Sep 08 15:17:40.088253 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:40.088220    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8" event={"ID":"a7d5fc44-22a8-41fd-9178-6429f54dd5f8","Type":"ContainerStarted","Data":"44fce72541cd477912c24e78e063a1b90aa93148ec9ec3d43ceefc14c22d2baa"}
Sep 08 15:17:40.089755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:40.089730    2568 generic.go:356] "Generic (PLEG): container finished" podID="282a660b-1fad-49c7-a37c-c16266440f70" containerID="f7033c9091e6adf9c74b9d916862a0dc4b454c6a377f70a7a31c9f4f5789d7a9" exitCode=0
Sep 08 15:17:40.089875 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:40.089804    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk" event={"ID":"282a660b-1fad-49c7-a37c-c16266440f70","Type":"ContainerDied","Data":"f7033c9091e6adf9c74b9d916862a0dc4b454c6a377f70a7a31c9f4f5789d7a9"}
Sep 08 15:17:40.089875 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:40.089837    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk" event={"ID":"282a660b-1fad-49c7-a37c-c16266440f70","Type":"ContainerStarted","Data":"d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa"}
Sep 08 15:17:40.090226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:40.090211    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:17:44.111176 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:44.111138    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8" event={"ID":"a7d5fc44-22a8-41fd-9178-6429f54dd5f8","Type":"ContainerStarted","Data":"60947af40fda40be34a302dd98560f7ad78d409dccc414112595f945a55e0b04"}
Sep 08 15:17:44.112952 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:44.112927    2568 generic.go:356] "Generic (PLEG): container finished" podID="282a660b-1fad-49c7-a37c-c16266440f70" containerID="091194a4b46d1140b98d58358c0d37df07d129781571b47fbfe19bcecad2218c" exitCode=0
Sep 08 15:17:44.113070 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:44.112961    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk" event={"ID":"282a660b-1fad-49c7-a37c-c16266440f70","Type":"ContainerDied","Data":"091194a4b46d1140b98d58358c0d37df07d129781571b47fbfe19bcecad2218c"}
Sep 08 15:17:45.124414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:45.124379    2568 generic.go:356] "Generic (PLEG): container finished" podID="a7d5fc44-22a8-41fd-9178-6429f54dd5f8" containerID="60947af40fda40be34a302dd98560f7ad78d409dccc414112595f945a55e0b04" exitCode=0
Sep 08 15:17:45.124861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:45.124459    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8" event={"ID":"a7d5fc44-22a8-41fd-9178-6429f54dd5f8","Type":"ContainerDied","Data":"60947af40fda40be34a302dd98560f7ad78d409dccc414112595f945a55e0b04"}
Sep 08 15:17:45.126661 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:45.126639    2568 generic.go:356] "Generic (PLEG): container finished" podID="282a660b-1fad-49c7-a37c-c16266440f70" containerID="ebb0677d039b68129a5710f5c417fe330ed537a19ac2971de0256b73dd9a7a4b" exitCode=0
Sep 08 15:17:45.126764 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:45.126722    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk" event={"ID":"282a660b-1fad-49c7-a37c-c16266440f70","Type":"ContainerDied","Data":"ebb0677d039b68129a5710f5c417fe330ed537a19ac2971de0256b73dd9a7a4b"}
Sep 08 15:17:46.258876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.258854    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:46.301757 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.301727    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjhkw\" (UniqueName: \"kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw\") pod \"282a660b-1fad-49c7-a37c-c16266440f70\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") "
Sep 08 15:17:46.301887 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.301784    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle\") pod \"282a660b-1fad-49c7-a37c-c16266440f70\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") "
Sep 08 15:17:46.301987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.301941    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util\") pod \"282a660b-1fad-49c7-a37c-c16266440f70\" (UID: \"282a660b-1fad-49c7-a37c-c16266440f70\") "
Sep 08 15:17:46.302487 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.302462    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle" (OuterVolumeSpecName: "bundle") pod "282a660b-1fad-49c7-a37c-c16266440f70" (UID: "282a660b-1fad-49c7-a37c-c16266440f70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:46.303762 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.303740    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw" (OuterVolumeSpecName: "kube-api-access-pjhkw") pod "282a660b-1fad-49c7-a37c-c16266440f70" (UID: "282a660b-1fad-49c7-a37c-c16266440f70"). InnerVolumeSpecName "kube-api-access-pjhkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:17:46.307230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.307188    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util" (OuterVolumeSpecName: "util") pod "282a660b-1fad-49c7-a37c-c16266440f70" (UID: "282a660b-1fad-49c7-a37c-c16266440f70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:46.402993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.402937    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhkw\" (UniqueName: \"kubernetes.io/projected/282a660b-1fad-49c7-a37c-c16266440f70-kube-api-access-pjhkw\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:46.402993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.402960    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:46.402993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:46.402969    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/282a660b-1fad-49c7-a37c-c16266440f70-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:17:47.137253 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:47.137225    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk"
Sep 08 15:17:47.137458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:47.137221    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0kxfgk" event={"ID":"282a660b-1fad-49c7-a37c-c16266440f70","Type":"ContainerDied","Data":"d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa"}
Sep 08 15:17:47.137458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:47.137361    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d696c96069a315367dfb3b74739dab69cd069e7875eb37f15c3e2480ea2797fa"
Sep 08 15:17:48.146983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:48.146947    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8" event={"ID":"a7d5fc44-22a8-41fd-9178-6429f54dd5f8","Type":"ContainerStarted","Data":"e3f93a0f47bdda46fa030d92bf86cad9b1e90d7df5bbe5ae0e27c2ef21601a8b"}
Sep 08 15:17:48.147450 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:48.147084    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:17:48.180813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:48.180754    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8" podStartSLOduration=3.187958971 podStartE2EDuration="10.180732662s" totalImagesPullingTime="5.87847275s" totalInitContainerRuntime="1.114300941s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:48.179509196 +0000 UTC m=+310.022306420" watchObservedRunningTime="2026-09-08 15:17:48.180732662 +0000 UTC m=+310.023529886"
Sep 08 15:17:59.154381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:17:59.154346    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68dbb6d945-qn7m8"
Sep 08 15:18:01.934804 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.934771    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"]
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935081    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="extract"
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935093    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="extract"
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935102    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="util"
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935107    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="util"
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935117    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="pull"
Sep 08 15:18:01.935177 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935122    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="pull"
Sep 08 15:18:01.935380 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.935193    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="282a660b-1fad-49c7-a37c-c16266440f70" containerName="extract"
Sep 08 15:18:01.937995 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.937978    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:01.941402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.941378    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sx25d\""
Sep 08 15:18:01.941510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.941399    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 15:18:01.941510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.941384    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 15:18:01.948415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:01.948386    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"]
Sep 08 15:18:02.027581 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.027550    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e24b30-1d08-4f99-aaed-339b7fb97346-tmp\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.027746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.027605    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4xj\" (UniqueName: \"kubernetes.io/projected/08e24b30-1d08-4f99-aaed-339b7fb97346-kube-api-access-ks4xj\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.027746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.027634    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e24b30-1d08-4f99-aaed-339b7fb97346-tls-certs\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.129022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.128979    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e24b30-1d08-4f99-aaed-339b7fb97346-tmp\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.129189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.129125    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4xj\" (UniqueName: \"kubernetes.io/projected/08e24b30-1d08-4f99-aaed-339b7fb97346-kube-api-access-ks4xj\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.129189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.129155    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e24b30-1d08-4f99-aaed-339b7fb97346-tls-certs\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.131427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.131398    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e24b30-1d08-4f99-aaed-339b7fb97346-tmp\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.131658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.131636    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08e24b30-1d08-4f99-aaed-339b7fb97346-tls-certs\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.141331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.141308    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4xj\" (UniqueName: \"kubernetes.io/projected/08e24b30-1d08-4f99-aaed-339b7fb97346-kube-api-access-ks4xj\") pod \"kube-auth-proxy-565756cbb6-tp4jn\" (UID: \"08e24b30-1d08-4f99-aaed-339b7fb97346\") " pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.248280 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.248192    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"
Sep 08 15:18:02.526942 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:02.526860    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn"]
Sep 08 15:18:02.529698 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:18:02.529672    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e24b30_1d08_4f99_aaed_339b7fb97346.slice/crio-52e29b83c484783e87b6b76c70eeae7ffdbe36705e6e098f6a207b3b6edc98b4 WatchSource:0}: Error finding container 52e29b83c484783e87b6b76c70eeae7ffdbe36705e6e098f6a207b3b6edc98b4: Status 404 returned error can't find the container with id 52e29b83c484783e87b6b76c70eeae7ffdbe36705e6e098f6a207b3b6edc98b4
Sep 08 15:18:03.057588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.057551    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:03.062092 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.062075    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.064608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.064580    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 15:18:03.064758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.064635    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g5rf5\""
Sep 08 15:18:03.064758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.064669    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 15:18:03.064870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.064829    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 15:18:03.064870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.064830    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 15:18:03.079574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.079548    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:03.138308 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138315    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138391    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138424    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wclq6\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138509    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138535    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.138555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.138552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.218882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.218846    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn" event={"ID":"08e24b30-1d08-4f99-aaed-339b7fb97346","Type":"ContainerStarted","Data":"52e29b83c484783e87b6b76c70eeae7ffdbe36705e6e098f6a207b3b6edc98b4"}
Sep 08 15:18:03.239276 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239246    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239276 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239280    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wclq6\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239342    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239364    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239381    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239401    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239490 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239424    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.239925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.239897    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.241674 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.241642    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.241866 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.241848    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.241913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.241895    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.242004 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.241982    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.248066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.248037    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wclq6\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.248152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.248107    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zhwv7\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.377990 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.377962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:03.655932 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:03.655849    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:03.658325 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:18:03.658286    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969bb618_9ed1_4180_b115_8c62a972cd99.slice/crio-9b3bde53239ed9e84f9d71bcf08004bbc88fbbc0fe5649669abee9fc104d5a8d WatchSource:0}: Error finding container 9b3bde53239ed9e84f9d71bcf08004bbc88fbbc0fe5649669abee9fc104d5a8d: Status 404 returned error can't find the container with id 9b3bde53239ed9e84f9d71bcf08004bbc88fbbc0fe5649669abee9fc104d5a8d
Sep 08 15:18:04.229573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:04.229530    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" event={"ID":"969bb618-9ed1-4180-b115-8c62a972cd99","Type":"ContainerStarted","Data":"9b3bde53239ed9e84f9d71bcf08004bbc88fbbc0fe5649669abee9fc104d5a8d"}
Sep 08 15:18:07.035062 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.035021    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 08 15:18:07.035347 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.035095    2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 08 15:18:07.248666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.248628    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn" event={"ID":"08e24b30-1d08-4f99-aaed-339b7fb97346","Type":"ContainerStarted","Data":"f0b8ff39e3a35ba472f1ff6c7890c768c379fcd4c4223a7a566cebb35cb6e056"}
Sep 08 15:18:07.250445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.250419    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" event={"ID":"969bb618-9ed1-4180-b115-8c62a972cd99","Type":"ContainerStarted","Data":"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"}
Sep 08 15:18:07.250569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.250542    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:07.267239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.267187    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-565756cbb6-tp4jn" podStartSLOduration=1.759798966 podStartE2EDuration="6.267167036s" totalImagesPullingTime="4.50736807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:07.264906886 +0000 UTC m=+329.107704108" watchObservedRunningTime="2026-09-08 15:18:07.267167036 +0000 UTC m=+329.109964260"
Sep 08 15:18:07.290701 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:07.290607    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" podStartSLOduration=0.916165591 podStartE2EDuration="4.290586165s" totalImagesPullingTime="3.374420574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:07.287860893 +0000 UTC m=+329.130658117" watchObservedRunningTime="2026-09-08 15:18:07.290586165 +0000 UTC m=+329.133383392"
Sep 08 15:18:08.257029 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:08.256995    2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zhwv7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 08 15:18:08.257475 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:08.257057    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 15:18:11.256785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:11.256756    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:23.272768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.272727    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"]
Sep 08 15:18:23.283068 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.283039    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.286286 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286259    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lzgpf\""
Sep 08 15:18:23.286561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286361    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 15:18:23.286687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286386    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 15:18:23.286687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286386    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:18:23.286687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286403    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 15:18:23.286687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.286459    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 15:18:23.287988 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.287967    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"]
Sep 08 15:18:23.415674 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.415643    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d11219a-26ae-49c1-84d0-b879904036ef-manager-config\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.415859 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.415692    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmg4t\" (UniqueName: \"kubernetes.io/projected/1d11219a-26ae-49c1-84d0-b879904036ef-kube-api-access-vmg4t\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.415859 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.415717    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.415859 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.415811    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-metrics-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.516938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.516910    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d11219a-26ae-49c1-84d0-b879904036ef-manager-config\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.517086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.516957    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmg4t\" (UniqueName: \"kubernetes.io/projected/1d11219a-26ae-49c1-84d0-b879904036ef-kube-api-access-vmg4t\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.517086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.516987    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.517086 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.517036    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-metrics-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.517648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.517620    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d11219a-26ae-49c1-84d0-b879904036ef-manager-config\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.519541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.519515    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.519624 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.519607    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d11219a-26ae-49c1-84d0-b879904036ef-metrics-cert\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.528693 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.528644    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmg4t\" (UniqueName: \"kubernetes.io/projected/1d11219a-26ae-49c1-84d0-b879904036ef-kube-api-access-vmg4t\") pod \"lws-controller-manager-7c78744898-gd52z\" (UID: \"1d11219a-26ae-49c1-84d0-b879904036ef\") " pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.594479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.594455    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:23.898826 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:23.898784    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"]
Sep 08 15:18:24.335549 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:24.335461    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z" event={"ID":"1d11219a-26ae-49c1-84d0-b879904036ef","Type":"ContainerStarted","Data":"501fec97e197ea9a0195c1d9644799805f8699baa594a0f28007e31776b9cbbb"}
Sep 08 15:18:24.637602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:24.637573    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:24.637822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:24.637777    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" containerName="discovery" containerID="cri-o://23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad" gracePeriod=30
Sep 08 15:18:24.889730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:24.889663    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:25.032801 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.032763    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.032981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.032812    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wclq6\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.032981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.032885    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.032981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.032914    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.032981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.032974    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.033196 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.033007    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.033196 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.033036    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token\") pod \"969bb618-9ed1-4180-b115-8c62a972cd99\" (UID: \"969bb618-9ed1-4180-b115-8c62a972cd99\") "
Sep 08 15:18:25.034032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.033896    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:18:25.035923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.035896    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token" (OuterVolumeSpecName: "istio-token") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:25.036103 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.036077    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6" (OuterVolumeSpecName: "kube-api-access-wclq6") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "kube-api-access-wclq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:25.036189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.036119    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:25.036189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.036130    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts" (OuterVolumeSpecName: "cacerts") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:25.036189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.036161    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs" (OuterVolumeSpecName: "local-certs") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:25.037164 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.037142    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "969bb618-9ed1-4180-b115-8c62a972cd99" (UID: "969bb618-9ed1-4180-b115-8c62a972cd99"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:25.134250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134216    2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/969bb618-9ed1-4180-b115-8c62a972cd99-local-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134249    2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-cacerts\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134261    2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-kubeconfig\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134274    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-ca-configmap\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134285    2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-istio-token\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134311    2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/969bb618-9ed1-4180-b115-8c62a972cd99-istio-csr-dns-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.134465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.134322    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wclq6\" (UniqueName: \"kubernetes.io/projected/969bb618-9ed1-4180-b115-8c62a972cd99-kube-api-access-wclq6\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:18:25.343329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.343237    2568 generic.go:356] "Generic (PLEG): container finished" podID="969bb618-9ed1-4180-b115-8c62a972cd99" containerID="23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad" exitCode=0
Sep 08 15:18:25.343329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.343308    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"
Sep 08 15:18:25.343842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.343332    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" event={"ID":"969bb618-9ed1-4180-b115-8c62a972cd99","Type":"ContainerDied","Data":"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"}
Sep 08 15:18:25.343842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.343387    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7" event={"ID":"969bb618-9ed1-4180-b115-8c62a972cd99","Type":"ContainerDied","Data":"9b3bde53239ed9e84f9d71bcf08004bbc88fbbc0fe5649669abee9fc104d5a8d"}
Sep 08 15:18:25.343842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.343412    2568 scope.go:117] "RemoveContainer" containerID="23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"
Sep 08 15:18:25.367003 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.366976    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:25.369827 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.369806    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zhwv7"]
Sep 08 15:18:25.705023 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.704834    2568 scope.go:117] "RemoveContainer" containerID="23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"
Sep 08 15:18:25.705210 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:18:25.705181    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad\": container with ID starting with 23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad not found: ID does not exist" containerID="23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"
Sep 08 15:18:25.705252 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:25.705222    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad"} err="failed to get container status \"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad\": rpc error: code = NotFound desc = could not find container \"23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad\": container with ID starting with 23da00c09e60b2fa4c7185eaa815aeb7dd2d4450d77fbb6cb2cadb394d3ecaad not found: ID does not exist"
Sep 08 15:18:26.349900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:26.349865    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z" event={"ID":"1d11219a-26ae-49c1-84d0-b879904036ef","Type":"ContainerStarted","Data":"8447d2c9d6f4bcd06bb23ed56b24177a3bb3af8ac8b3b65fd80b192118869a7e"}
Sep 08 15:18:26.350399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:26.349956    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:18:26.368302 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:26.368254    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z" podStartSLOduration=1.515579002 podStartE2EDuration="3.368241159s" totalImagesPullingTime="1.852662157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:26.36751522 +0000 UTC m=+348.210312444" watchObservedRunningTime="2026-09-08 15:18:26.368241159 +0000 UTC m=+348.211038381"
Sep 08 15:18:26.680301 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:26.680259    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" path="/var/lib/kubelet/pods/969bb618-9ed1-4180-b115-8c62a972cd99/volumes"
Sep 08 15:18:37.358351 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:18:37.358257    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c78744898-gd52z"
Sep 08 15:19:16.173730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.173694    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"]
Sep 08 15:19:16.174126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.174011    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" containerName="discovery"
Sep 08 15:19:16.174126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.174022    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" containerName="discovery"
Sep 08 15:19:16.174126 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.174081    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="969bb618-9ed1-4180-b115-8c62a972cd99" containerName="discovery"
Sep 08 15:19:16.177272 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.177255    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.179699 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.179674    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 08 15:19:16.179832 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.179719    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 08 15:19:16.180366 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.180350    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rprqj\""
Sep 08 15:19:16.183641 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.183617    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"]
Sep 08 15:19:16.271082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.271054    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"]
Sep 08 15:19:16.274511 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.274492    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.283137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.283114    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"]
Sep 08 15:19:16.346887 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.346863    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.347006 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.346902    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.347006 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.346932    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8j2\" (UniqueName: \"kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.372921 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.372897    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"]
Sep 08 15:19:16.376358 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.376343    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.384307 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.384276    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"]
Sep 08 15:19:16.447627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447561    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.447627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447600    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8j2\" (UniqueName: \"kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.447769 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447636    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.447769 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9cp\" (UniqueName: \"kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.447769 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447700    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.447880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447775    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.447963 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.447945    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.448061 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.448046    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.455848 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.455822    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8j2\" (UniqueName: \"kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.472667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.472644    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"]
Sep 08 15:19:16.476159 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.476145    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.484485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.484463    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"]
Sep 08 15:19:16.487429 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.487411    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:16.549054 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549014    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.549180 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549162    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.549248 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549196    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrndx\" (UniqueName: \"kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.549248 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549227    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9cp\" (UniqueName: \"kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.549403 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549254    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.549403 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.549303    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.550235 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.550204    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.550492 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.550465    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.559317 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.558543    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9cp\" (UniqueName: \"kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.583655 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.583631    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:16.649832 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.649792    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj7l\" (UniqueName: \"kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.649940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.649907    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.649983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.649947    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrndx\" (UniqueName: \"kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.649983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.649977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.650058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.650006    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.650058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.650040    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.650420 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.650401    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.650470 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.650428    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.659037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.659012    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrndx\" (UniqueName: \"kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.685898 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.685875    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:16.750821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.750789    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.750967 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.750856    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj7l\" (UniqueName: \"kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.750967 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.750896    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.751175 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.751156    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.751236 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.751179    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.759799 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.759747    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj7l\" (UniqueName: \"kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.762664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.762645    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"]
Sep 08 15:19:16.764792 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:19:16.764767    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20d371e_cdeb_497e_a7f8_ebd392b572f5.slice/crio-db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d WatchSource:0}: Error finding container db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d: Status 404 returned error can't find the container with id db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d
Sep 08 15:19:16.786651 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.786629    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:16.856132 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.856104    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"]
Sep 08 15:19:16.858578 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:19:16.858547    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80566fcd_fde7_4a17_9d33_7a383422dabb.slice/crio-6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578 WatchSource:0}: Error finding container 6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578: Status 404 returned error can't find the container with id 6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578
Sep 08 15:19:16.958491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:16.958469    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"]
Sep 08 15:19:16.960159 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:19:16.960130    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5a1cb3_b410_4926_b4be_1e782324c44b.slice/crio-a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8 WatchSource:0}: Error finding container a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8: Status 404 returned error can't find the container with id a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8
Sep 08 15:19:17.059112 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.059090    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"]
Sep 08 15:19:17.067378 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:19:17.067352    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bf6154_7ef6_449d_a80d_30cea8802083.slice/crio-3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef WatchSource:0}: Error finding container 3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef: Status 404 returned error can't find the container with id 3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef
Sep 08 15:19:17.588372 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.588337    2568 generic.go:356] "Generic (PLEG): container finished" podID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerID="1d8f538c08d5d2dc811646cc6511f00bdce717a4aa73cc011c05ffd3a4877335" exitCode=0
Sep 08 15:19:17.588831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.588421    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerDied","Data":"1d8f538c08d5d2dc811646cc6511f00bdce717a4aa73cc011c05ffd3a4877335"}
Sep 08 15:19:17.588831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.588484    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerStarted","Data":"a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8"}
Sep 08 15:19:17.590034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.589986    2568 generic.go:356] "Generic (PLEG): container finished" podID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerID="8aed93b882e146e99774c16663960a22cde6cbb73d2a4a70db3350780839e5e9" exitCode=0
Sep 08 15:19:17.590085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.590063    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f" event={"ID":"80566fcd-fde7-4a17-9d33-7a383422dabb","Type":"ContainerDied","Data":"8aed93b882e146e99774c16663960a22cde6cbb73d2a4a70db3350780839e5e9"}
Sep 08 15:19:17.590141 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.590106    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f" event={"ID":"80566fcd-fde7-4a17-9d33-7a383422dabb","Type":"ContainerStarted","Data":"6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578"}
Sep 08 15:19:17.591702 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.591685    2568 generic.go:356] "Generic (PLEG): container finished" podID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerID="f2bd75b276f15543ff9def72689b4e611e245e9ec822c699974842d6149b1e80" exitCode=0
Sep 08 15:19:17.591785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.591764    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerDied","Data":"f2bd75b276f15543ff9def72689b4e611e245e9ec822c699974842d6149b1e80"}
Sep 08 15:19:17.591842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.591809    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerStarted","Data":"3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef"}
Sep 08 15:19:17.593419 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.593401    2568 generic.go:356] "Generic (PLEG): container finished" podID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerID="4036c2d7279ae78e5d68f059bf76b882f4085627d22e505550abe08fbbc62386" exitCode=0
Sep 08 15:19:17.593510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.593463    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882" event={"ID":"a20d371e-cdeb-497e-a7f8-ebd392b572f5","Type":"ContainerDied","Data":"4036c2d7279ae78e5d68f059bf76b882f4085627d22e505550abe08fbbc62386"}
Sep 08 15:19:17.593510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:17.593489    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882" event={"ID":"a20d371e-cdeb-497e-a7f8-ebd392b572f5","Type":"ContainerStarted","Data":"db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d"}
Sep 08 15:19:18.602576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:18.602540    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerStarted","Data":"510300c537c00c7107efedc4e554303bd6ac1bd5b2da731bcb2bc69e63696f2c"}
Sep 08 15:19:18.604829 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:18.604801    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerStarted","Data":"933fa34069b70de5f48aa6f528a282e298e087479f63fc0e3668842c424823e1"}
Sep 08 15:19:19.612688 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.612652    2568 generic.go:356] "Generic (PLEG): container finished" podID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerID="510300c537c00c7107efedc4e554303bd6ac1bd5b2da731bcb2bc69e63696f2c" exitCode=0
Sep 08 15:19:19.613081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.612729    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerDied","Data":"510300c537c00c7107efedc4e554303bd6ac1bd5b2da731bcb2bc69e63696f2c"}
Sep 08 15:19:19.614785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.614763    2568 generic.go:356] "Generic (PLEG): container finished" podID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerID="3d11aeae9159bd1052539d6f744395a91b0888841678f4332bf36d092861aeea" exitCode=0
Sep 08 15:19:19.614899 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.614847    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f" event={"ID":"80566fcd-fde7-4a17-9d33-7a383422dabb","Type":"ContainerDied","Data":"3d11aeae9159bd1052539d6f744395a91b0888841678f4332bf36d092861aeea"}
Sep 08 15:19:19.616874 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.616853    2568 generic.go:356] "Generic (PLEG): container finished" podID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerID="933fa34069b70de5f48aa6f528a282e298e087479f63fc0e3668842c424823e1" exitCode=0
Sep 08 15:19:19.616976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.616936    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerDied","Data":"933fa34069b70de5f48aa6f528a282e298e087479f63fc0e3668842c424823e1"}
Sep 08 15:19:19.622388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.622368    2568 generic.go:356] "Generic (PLEG): container finished" podID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerID="eb37aaf1b14efbc25bd3298ee28236a8339f8a56b5f6c22422c2400034a96ace" exitCode=0
Sep 08 15:19:19.622500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:19.622397    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882" event={"ID":"a20d371e-cdeb-497e-a7f8-ebd392b572f5","Type":"ContainerDied","Data":"eb37aaf1b14efbc25bd3298ee28236a8339f8a56b5f6c22422c2400034a96ace"}
Sep 08 15:19:20.629233 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.629195    2568 generic.go:356] "Generic (PLEG): container finished" podID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerID="91c8d57ce5a5c0e164f0c184c9f17974bd2e857ac2ae88326e167f8fbd055b4d" exitCode=0
Sep 08 15:19:20.629639 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.629288    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerDied","Data":"91c8d57ce5a5c0e164f0c184c9f17974bd2e857ac2ae88326e167f8fbd055b4d"}
Sep 08 15:19:20.631338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.631319    2568 generic.go:356] "Generic (PLEG): container finished" podID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerID="48957f05486a7d602605f50ccb6853d725244e4617d51ecfd2912feeb0758e1f" exitCode=0
Sep 08 15:19:20.631428 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.631412    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f" event={"ID":"80566fcd-fde7-4a17-9d33-7a383422dabb","Type":"ContainerDied","Data":"48957f05486a7d602605f50ccb6853d725244e4617d51ecfd2912feeb0758e1f"}
Sep 08 15:19:20.633550 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.633525    2568 generic.go:356] "Generic (PLEG): container finished" podID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerID="84705e7c32ae33dd20eeef1f0f76e0517bfa1e27a5a801486ff01bc9a724e004" exitCode=0
Sep 08 15:19:20.633659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.633614    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerDied","Data":"84705e7c32ae33dd20eeef1f0f76e0517bfa1e27a5a801486ff01bc9a724e004"}
Sep 08 15:19:20.635522 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.635504    2568 generic.go:356] "Generic (PLEG): container finished" podID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerID="5e8b2246300e66f25fbc9a4a165bafe5db2e8c6940279ab9da0113b9c7fb7290" exitCode=0
Sep 08 15:19:20.635604 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:20.635530    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882" event={"ID":"a20d371e-cdeb-497e-a7f8-ebd392b572f5","Type":"ContainerDied","Data":"5e8b2246300e66f25fbc9a4a165bafe5db2e8c6940279ab9da0113b9c7fb7290"}
Sep 08 15:19:21.828920 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.828881    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:21.832813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.832792    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:21.836194 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.836171    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:21.839639 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.839622    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:21.998538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998458    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util\") pod \"b7bf6154-7ef6-449d-a80d-30cea8802083\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") "
Sep 08 15:19:21.998538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998507    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle\") pod \"b7bf6154-7ef6-449d-a80d-30cea8802083\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998550    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle\") pod \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998580    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle\") pod \"dd5a1cb3-b410-4926-b4be-1e782324c44b\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998610    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util\") pod \"80566fcd-fde7-4a17-9d33-7a383422dabb\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998641    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw8j2\" (UniqueName: \"kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2\") pod \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998683    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle\") pod \"80566fcd-fde7-4a17-9d33-7a383422dabb\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998708    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util\") pod \"dd5a1cb3-b410-4926-b4be-1e782324c44b\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") "
Sep 08 15:19:21.998744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998744    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrndx\" (UniqueName: \"kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx\") pod \"dd5a1cb3-b410-4926-b4be-1e782324c44b\" (UID: \"dd5a1cb3-b410-4926-b4be-1e782324c44b\") "
Sep 08 15:19:21.999083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998779    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzj7l\" (UniqueName: \"kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l\") pod \"b7bf6154-7ef6-449d-a80d-30cea8802083\" (UID: \"b7bf6154-7ef6-449d-a80d-30cea8802083\") "
Sep 08 15:19:21.999083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998827    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util\") pod \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\" (UID: \"a20d371e-cdeb-497e-a7f8-ebd392b572f5\") "
Sep 08 15:19:21.999083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.998850    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj9cp\" (UniqueName: \"kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp\") pod \"80566fcd-fde7-4a17-9d33-7a383422dabb\" (UID: \"80566fcd-fde7-4a17-9d33-7a383422dabb\") "
Sep 08 15:19:21.999239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999081    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle" (OuterVolumeSpecName: "bundle") pod "a20d371e-cdeb-497e-a7f8-ebd392b572f5" (UID: "a20d371e-cdeb-497e-a7f8-ebd392b572f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:21.999239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999080    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle" (OuterVolumeSpecName: "bundle") pod "b7bf6154-7ef6-449d-a80d-30cea8802083" (UID: "b7bf6154-7ef6-449d-a80d-30cea8802083"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:21.999239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999159    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle" (OuterVolumeSpecName: "bundle") pod "dd5a1cb3-b410-4926-b4be-1e782324c44b" (UID: "dd5a1cb3-b410-4926-b4be-1e782324c44b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:21.999962 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999936    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:21.999962 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999963    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.000138 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:21.999975    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.000195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.000174    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle" (OuterVolumeSpecName: "bundle") pod "80566fcd-fde7-4a17-9d33-7a383422dabb" (UID: "80566fcd-fde7-4a17-9d33-7a383422dabb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:22.001773 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.001743    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l" (OuterVolumeSpecName: "kube-api-access-tzj7l") pod "b7bf6154-7ef6-449d-a80d-30cea8802083" (UID: "b7bf6154-7ef6-449d-a80d-30cea8802083"). InnerVolumeSpecName "kube-api-access-tzj7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:22.002259 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.002219    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2" (OuterVolumeSpecName: "kube-api-access-pw8j2") pod "a20d371e-cdeb-497e-a7f8-ebd392b572f5" (UID: "a20d371e-cdeb-497e-a7f8-ebd392b572f5"). InnerVolumeSpecName "kube-api-access-pw8j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:22.002367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.002340    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx" (OuterVolumeSpecName: "kube-api-access-hrndx") pod "dd5a1cb3-b410-4926-b4be-1e782324c44b" (UID: "dd5a1cb3-b410-4926-b4be-1e782324c44b"). InnerVolumeSpecName "kube-api-access-hrndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:22.002572 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.002548    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp" (OuterVolumeSpecName: "kube-api-access-nj9cp") pod "80566fcd-fde7-4a17-9d33-7a383422dabb" (UID: "80566fcd-fde7-4a17-9d33-7a383422dabb"). InnerVolumeSpecName "kube-api-access-nj9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:22.005971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.005943    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util" (OuterVolumeSpecName: "util") pod "a20d371e-cdeb-497e-a7f8-ebd392b572f5" (UID: "a20d371e-cdeb-497e-a7f8-ebd392b572f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:22.006461 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.006443    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util" (OuterVolumeSpecName: "util") pod "dd5a1cb3-b410-4926-b4be-1e782324c44b" (UID: "dd5a1cb3-b410-4926-b4be-1e782324c44b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:22.007210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.007189    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util" (OuterVolumeSpecName: "util") pod "b7bf6154-7ef6-449d-a80d-30cea8802083" (UID: "b7bf6154-7ef6-449d-a80d-30cea8802083"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:22.007741 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.007725    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util" (OuterVolumeSpecName: "util") pod "80566fcd-fde7-4a17-9d33-7a383422dabb" (UID: "80566fcd-fde7-4a17-9d33-7a383422dabb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:19:22.101034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101011    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101030    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw8j2\" (UniqueName: \"kubernetes.io/projected/a20d371e-cdeb-497e-a7f8-ebd392b572f5-kube-api-access-pw8j2\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101044    2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80566fcd-fde7-4a17-9d33-7a383422dabb-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101052    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd5a1cb3-b410-4926-b4be-1e782324c44b-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101060    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrndx\" (UniqueName: \"kubernetes.io/projected/dd5a1cb3-b410-4926-b4be-1e782324c44b-kube-api-access-hrndx\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101068    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj7l\" (UniqueName: \"kubernetes.io/projected/b7bf6154-7ef6-449d-a80d-30cea8802083-kube-api-access-tzj7l\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101078    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a20d371e-cdeb-497e-a7f8-ebd392b572f5-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101087    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9cp\" (UniqueName: \"kubernetes.io/projected/80566fcd-fde7-4a17-9d33-7a383422dabb-kube-api-access-nj9cp\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.101165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.101095    2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7bf6154-7ef6-449d-a80d-30cea8802083-util\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:22.647864 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.647830    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx" event={"ID":"dd5a1cb3-b410-4926-b4be-1e782324c44b","Type":"ContainerDied","Data":"a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8"}
Sep 08 15:19:22.648042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.647880    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9deca9be4cb37dd27738b09817e0b0366145f3ba53cf02c52239099328c36e8"
Sep 08 15:19:22.648042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.647836    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ch4wqx"
Sep 08 15:19:22.649861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.649830    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f"
Sep 08 15:19:22.649861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.649838    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053j4l2f" event={"ID":"80566fcd-fde7-4a17-9d33-7a383422dabb","Type":"ContainerDied","Data":"6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578"}
Sep 08 15:19:22.650035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.649877    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be056a0c8564bb73b650780705dbc0fefeaafed619e83e1beec1fe99fbdf578"
Sep 08 15:19:22.652056 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.652038    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf"
Sep 08 15:19:22.652056 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.652041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0nkzsf" event={"ID":"b7bf6154-7ef6-449d-a80d-30cea8802083","Type":"ContainerDied","Data":"3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef"}
Sep 08 15:19:22.652240 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.652085    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c63bcc90c3432f74cc11aff73e16f15aedebb9011ec337a78e8ef7ca8264cef"
Sep 08 15:19:22.654073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.654052    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882" event={"ID":"a20d371e-cdeb-497e-a7f8-ebd392b572f5","Type":"ContainerDied","Data":"db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d"}
Sep 08 15:19:22.654176 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.654086    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6e194892494785c31fc0decb8be5cfbf1fb83a943e6096cabb7865a94dce2d"
Sep 08 15:19:22.654176 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:22.654085    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbrv882"
Sep 08 15:19:32.715070 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:32.715032    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:19:42.710566 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710533    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"]
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710871    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710883    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710890    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710896    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710901    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="extract"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710907    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="extract"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710917    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710925    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="util"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710934    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="pull"
Sep 08 15:19:42.710945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710940    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710958    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710963    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710972    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710977    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710985    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="util"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710991    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="util"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.710999    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711007    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711020    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711025    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711030    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711035    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711043    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711048    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="pull"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711101    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5a1cb3-b410-4926-b4be-1e782324c44b" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711110    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="80566fcd-fde7-4a17-9d33-7a383422dabb" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711116    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a20d371e-cdeb-497e-a7f8-ebd392b572f5" containerName="extract"
Sep 08 15:19:42.711268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.711123    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7bf6154-7ef6-449d-a80d-30cea8802083" containerName="extract"
Sep 08 15:19:42.715660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.715633    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.719392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.718497    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 15:19:42.719392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.718592    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 15:19:42.719392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.718893    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 15:19:42.719623 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.719453    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gsk2s\""
Sep 08 15:19:42.719964 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.719728    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 15:19:42.723806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.723782    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"]
Sep 08 15:19:42.746533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.746508    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e139d92-d50e-4c9e-a760-602dc4b51924-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.746676 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.746554    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.746676 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.746624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42t9c\" (UniqueName: \"kubernetes.io/projected/3e139d92-d50e-4c9e-a760-602dc4b51924-kube-api-access-42t9c\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.847752 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.847718    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.847913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.847766    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42t9c\" (UniqueName: \"kubernetes.io/projected/3e139d92-d50e-4c9e-a760-602dc4b51924-kube-api-access-42t9c\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.847913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.847801    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e139d92-d50e-4c9e-a760-602dc4b51924-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.847913 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:19:42.847870    2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 08 15:19:42.848085 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:19:42.847959    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert podName:3e139d92-d50e-4c9e-a760-602dc4b51924 nodeName:}" failed. No retries permitted until 2026-09-08 15:19:43.347936973 +0000 UTC m=+425.190734187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-gkcnl" (UID: "3e139d92-d50e-4c9e-a760-602dc4b51924") : secret "plugin-serving-cert" not found
Sep 08 15:19:42.848479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.848461    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3e139d92-d50e-4c9e-a760-602dc4b51924-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:42.857048 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:42.857025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42t9c\" (UniqueName: \"kubernetes.io/projected/3e139d92-d50e-4c9e-a760-602dc4b51924-kube-api-access-42t9c\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:43.352087 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:43.352053    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:43.354443 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:43.354419    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3e139d92-d50e-4c9e-a760-602dc4b51924-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gkcnl\" (UID: \"3e139d92-d50e-4c9e-a760-602dc4b51924\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:43.628987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:43.628962    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"
Sep 08 15:19:43.922866 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:43.922834    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl"]
Sep 08 15:19:43.924325 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:19:43.924284    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e139d92_d50e_4c9e_a760_602dc4b51924.slice/crio-90a9491363e94715ed61fe7b54a2d30aed10c7ef80b17283c22fd14406a39914 WatchSource:0}: Error finding container 90a9491363e94715ed61fe7b54a2d30aed10c7ef80b17283c22fd14406a39914: Status 404 returned error can't find the container with id 90a9491363e94715ed61fe7b54a2d30aed10c7ef80b17283c22fd14406a39914
Sep 08 15:19:44.762724 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:44.762681    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl" event={"ID":"3e139d92-d50e-4c9e-a760-602dc4b51924","Type":"ContainerStarted","Data":"90a9491363e94715ed61fe7b54a2d30aed10c7ef80b17283c22fd14406a39914"}
Sep 08 15:19:48.791441 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:48.791400    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl" event={"ID":"3e139d92-d50e-4c9e-a760-602dc4b51924","Type":"ContainerStarted","Data":"3fbc8cbf9fabe8e7222fbee4a606364b628d81c5a123f87e15c985407003d4e9"}
Sep 08 15:19:48.815676 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:48.815630    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gkcnl" podStartSLOduration=2.098667456 podStartE2EDuration="6.815615391s" totalImagesPullingTime="4.716947935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:48.814094292 +0000 UTC m=+430.656891514" watchObservedRunningTime="2026-09-08 15:19:48.815615391 +0000 UTC m=+430.658412619"
Sep 08 15:19:57.736185 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:57.736124    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f77845f55-bh5nq" podUID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" containerName="console" containerID="cri-o://7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a" gracePeriod=15
Sep 08 15:19:57.983889 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:57.983860    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:19:58.076242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076174    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-478l7\" (UniqueName: \"kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076233    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076252    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076286    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076328    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076375    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076404    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca\") pod \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\" (UID: \"58347eb5-0ba1-4d63-9029-47ba6f4372a0\") "
Sep 08 15:19:58.076793 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076735    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:19:58.076793 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076771    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config" (OuterVolumeSpecName: "console-config") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:19:58.076919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076868    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:19:58.076919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.076874    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:19:58.078524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.078498    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:19:58.078636 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.078559    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7" (OuterVolumeSpecName: "kube-api-access-478l7") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "kube-api-access-478l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:58.078636 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.078587    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "58347eb5-0ba1-4d63-9029-47ba6f4372a0" (UID: "58347eb5-0ba1-4d63-9029-47ba6f4372a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:19:58.177118 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177087    2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-oauth-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177118 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177112    2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-config\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177118 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177122    2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-oauth-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177341 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177131    2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-trusted-ca-bundle\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177341 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177139    2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58347eb5-0ba1-4d63-9029-47ba6f4372a0-console-serving-cert\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177341 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177148    2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58347eb5-0ba1-4d63-9029-47ba6f4372a0-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.177341 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.177156    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478l7\" (UniqueName: \"kubernetes.io/projected/58347eb5-0ba1-4d63-9029-47ba6f4372a0-kube-api-access-478l7\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:19:58.838882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.838845    2568 generic.go:356] "Generic (PLEG): container finished" podID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" containerID="7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a" exitCode=0
Sep 08 15:19:58.839279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.838894    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f77845f55-bh5nq"
Sep 08 15:19:58.839279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.838929    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f77845f55-bh5nq" event={"ID":"58347eb5-0ba1-4d63-9029-47ba6f4372a0","Type":"ContainerDied","Data":"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"}
Sep 08 15:19:58.839279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.838984    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f77845f55-bh5nq" event={"ID":"58347eb5-0ba1-4d63-9029-47ba6f4372a0","Type":"ContainerDied","Data":"16223fba04a485c4cb89b6719ff663b50ba7c40960e7dd1be39fc658ba29bc6d"}
Sep 08 15:19:58.839279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.839008    2568 scope.go:117] "RemoveContainer" containerID="7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"
Sep 08 15:19:58.848790 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.848775    2568 scope.go:117] "RemoveContainer" containerID="7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"
Sep 08 15:19:58.848999 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:19:58.848980    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a\": container with ID starting with 7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a not found: ID does not exist" containerID="7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"
Sep 08 15:19:58.849043 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.849011    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a"} err="failed to get container status \"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a\": rpc error: code = NotFound desc = could not find container \"7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a\": container with ID starting with 7b8e615c6554679c8e45e240049869c43fc0714e711a8ce30c10797fd1ff054a not found: ID does not exist"
Sep 08 15:19:58.858223 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.858201    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:19:58.863305 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:19:58.863275    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f77845f55-bh5nq"]
Sep 08 15:20:00.675470 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:00.675439    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" path="/var/lib/kubelet/pods/58347eb5-0ba1-4d63-9029-47ba6f4372a0/volumes"
Sep 08 15:20:41.969024 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.968988    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:20:41.969446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.969321    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" containerName="console"
Sep 08 15:20:41.969446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.969335    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" containerName="console"
Sep 08 15:20:41.969446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.969406    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="58347eb5-0ba1-4d63-9029-47ba6f4372a0" containerName="console"
Sep 08 15:20:41.973302 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.973275    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:41.975573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.975557    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 15:20:41.980513 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:41.980488    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:20:42.009702 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.009676    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:20:42.026279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.026253    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhdt\" (UniqueName: \"kubernetes.io/projected/00b13e06-4bff-46f3-afdd-d24884457cd9-kube-api-access-gkhdt\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.026434 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.026314    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00b13e06-4bff-46f3-afdd-d24884457cd9-config-file\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.127366 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.127337    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhdt\" (UniqueName: \"kubernetes.io/projected/00b13e06-4bff-46f3-afdd-d24884457cd9-kube-api-access-gkhdt\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.127533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.127376    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00b13e06-4bff-46f3-afdd-d24884457cd9-config-file\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.127989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.127971    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/00b13e06-4bff-46f3-afdd-d24884457cd9-config-file\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.135612 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.135588    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhdt\" (UniqueName: \"kubernetes.io/projected/00b13e06-4bff-46f3-afdd-d24884457cd9-kube-api-access-gkhdt\") pod \"limitador-limitador-85d9db4d5d-bf5n7\" (UID: \"00b13e06-4bff-46f3-afdd-d24884457cd9\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.284597 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.284500    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:42.612397 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:20:42.612360    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b13e06_4bff_46f3_afdd_d24884457cd9.slice/crio-3bdabd42a66daa03a05a53e034660ae67cc7c8e2eadc8de5ed5defc2366fb351 WatchSource:0}: Error finding container 3bdabd42a66daa03a05a53e034660ae67cc7c8e2eadc8de5ed5defc2366fb351: Status 404 returned error can't find the container with id 3bdabd42a66daa03a05a53e034660ae67cc7c8e2eadc8de5ed5defc2366fb351
Sep 08 15:20:42.614400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.614284    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:20:42.789711 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.789684    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:20:42.794119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.794102    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:20:42.796278 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.796258    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bm7bp\""
Sep 08 15:20:42.799630 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.799606    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:20:42.833538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.833506    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dcm\" (UniqueName: \"kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm\") pod \"authorino-c548b5d7d-cqvlt\" (UID: \"e81d7a35-3920-4846-9fc5-f9fd842b0075\") " pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:20:42.934580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.934551    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dcm\" (UniqueName: \"kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm\") pod \"authorino-c548b5d7d-cqvlt\" (UID: \"e81d7a35-3920-4846-9fc5-f9fd842b0075\") " pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:20:42.943090 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:42.943068    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dcm\" (UniqueName: \"kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm\") pod \"authorino-c548b5d7d-cqvlt\" (UID: \"e81d7a35-3920-4846-9fc5-f9fd842b0075\") " pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:20:43.066500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:43.066465    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7" event={"ID":"00b13e06-4bff-46f3-afdd-d24884457cd9","Type":"ContainerStarted","Data":"3bdabd42a66daa03a05a53e034660ae67cc7c8e2eadc8de5ed5defc2366fb351"}
Sep 08 15:20:43.104616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:43.104588    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:20:43.375570 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:43.372452    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:20:43.376243 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:20:43.376220    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode81d7a35_3920_4846_9fc5_f9fd842b0075.slice/crio-e594a3d0ac3f23b81feec1356293fa0bda63a85c7a5983b18668b66507898ce9 WatchSource:0}: Error finding container e594a3d0ac3f23b81feec1356293fa0bda63a85c7a5983b18668b66507898ce9: Status 404 returned error can't find the container with id e594a3d0ac3f23b81feec1356293fa0bda63a85c7a5983b18668b66507898ce9
Sep 08 15:20:44.075247 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:44.075197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" event={"ID":"e81d7a35-3920-4846-9fc5-f9fd842b0075","Type":"ContainerStarted","Data":"e594a3d0ac3f23b81feec1356293fa0bda63a85c7a5983b18668b66507898ce9"}
Sep 08 15:20:46.088378 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:46.088342    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" event={"ID":"e81d7a35-3920-4846-9fc5-f9fd842b0075","Type":"ContainerStarted","Data":"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"}
Sep 08 15:20:46.089848 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:46.089826    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7" event={"ID":"00b13e06-4bff-46f3-afdd-d24884457cd9","Type":"ContainerStarted","Data":"85ddf67c76a76bbbc7d4906e9600f4315b2f533b6c6323b9afdfaf3cf6271c11"}
Sep 08 15:20:46.089975 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:46.089953    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:20:46.104500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:46.104454    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" podStartSLOduration=1.8983829719999998 podStartE2EDuration="4.104440448s" totalImagesPullingTime="2.206057476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:20:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:20:46.103061803 +0000 UTC m=+487.945859027" watchObservedRunningTime="2026-09-08 15:20:46.104440448 +0000 UTC m=+487.947237725"
Sep 08 15:20:46.119239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:46.119198    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7" podStartSLOduration=2.155994653 podStartE2EDuration="5.119183517s" totalImagesPullingTime="2.963188864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:20:46.118324334 +0000 UTC m=+487.961121558" watchObservedRunningTime="2026-09-08 15:20:46.119183517 +0000 UTC m=+487.961980739"
Sep 08 15:20:57.100521 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:20:57.100487    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"
Sep 08 15:21:16.039893 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.039810    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f564cd68b-zxvnh"]
Sep 08 15:21:16.045047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.045024    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.053409 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.053386    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f564cd68b-zxvnh"]
Sep 08 15:21:16.102493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.102468    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f564cd68b-zxvnh"]
Sep 08 15:21:16.102719 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:21:16.102692    2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-672cp], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7f564cd68b-zxvnh" podUID="1f6cbdbe-1aff-4805-976c-9c0918dfcc4a"
Sep 08 15:21:16.106573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.106549    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672cp\" (UniqueName: \"kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp\") pod \"authorino-7f564cd68b-zxvnh\" (UID: \"1f6cbdbe-1aff-4805-976c-9c0918dfcc4a\") " pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.206975 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.206944    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672cp\" (UniqueName: \"kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp\") pod \"authorino-7f564cd68b-zxvnh\" (UID: \"1f6cbdbe-1aff-4805-976c-9c0918dfcc4a\") " pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.214430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.214406    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-672cp\" (UniqueName: \"kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp\") pod \"authorino-7f564cd68b-zxvnh\" (UID: \"1f6cbdbe-1aff-4805-976c-9c0918dfcc4a\") " pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.243232 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.243205    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.250174 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.250157    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:16.408417 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.408389    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-672cp\" (UniqueName: \"kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp\") pod \"1f6cbdbe-1aff-4805-976c-9c0918dfcc4a\" (UID: \"1f6cbdbe-1aff-4805-976c-9c0918dfcc4a\") "
Sep 08 15:21:16.410521 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.410492    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp" (OuterVolumeSpecName: "kube-api-access-672cp") pod "1f6cbdbe-1aff-4805-976c-9c0918dfcc4a" (UID: "1f6cbdbe-1aff-4805-976c-9c0918dfcc4a"). InnerVolumeSpecName "kube-api-access-672cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:21:16.509614 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:16.509588    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672cp\" (UniqueName: \"kubernetes.io/projected/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a-kube-api-access-672cp\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:21:17.247989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:17.247954    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f564cd68b-zxvnh"
Sep 08 15:21:17.275934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:17.275902    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f564cd68b-zxvnh"]
Sep 08 15:21:17.279045 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:17.279020    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f564cd68b-zxvnh"]
Sep 08 15:21:18.300207 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.300171    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:21:18.300625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.300356    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" podUID="e81d7a35-3920-4846-9fc5-f9fd842b0075" containerName="authorino" containerID="cri-o://d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da" gracePeriod=30
Sep 08 15:21:18.537884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.537851    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:21:18.625849 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.625823    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8dcm\" (UniqueName: \"kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm\") pod \"e81d7a35-3920-4846-9fc5-f9fd842b0075\" (UID: \"e81d7a35-3920-4846-9fc5-f9fd842b0075\") "
Sep 08 15:21:18.628017 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.627988    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm" (OuterVolumeSpecName: "kube-api-access-s8dcm") pod "e81d7a35-3920-4846-9fc5-f9fd842b0075" (UID: "e81d7a35-3920-4846-9fc5-f9fd842b0075"). InnerVolumeSpecName "kube-api-access-s8dcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:21:18.675860 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.675830    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6cbdbe-1aff-4805-976c-9c0918dfcc4a" path="/var/lib/kubelet/pods/1f6cbdbe-1aff-4805-976c-9c0918dfcc4a/volumes"
Sep 08 15:21:18.726913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:18.726865    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8dcm\" (UniqueName: \"kubernetes.io/projected/e81d7a35-3920-4846-9fc5-f9fd842b0075-kube-api-access-s8dcm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:21:19.259567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.259533    2568 generic.go:356] "Generic (PLEG): container finished" podID="e81d7a35-3920-4846-9fc5-f9fd842b0075" containerID="d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da" exitCode=0
Sep 08 15:21:19.259817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.259576    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-cqvlt"
Sep 08 15:21:19.259817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.259586    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" event={"ID":"e81d7a35-3920-4846-9fc5-f9fd842b0075","Type":"ContainerDied","Data":"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"}
Sep 08 15:21:19.259817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.259624    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-cqvlt" event={"ID":"e81d7a35-3920-4846-9fc5-f9fd842b0075","Type":"ContainerDied","Data":"e594a3d0ac3f23b81feec1356293fa0bda63a85c7a5983b18668b66507898ce9"}
Sep 08 15:21:19.259817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.259646    2568 scope.go:117] "RemoveContainer" containerID="d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"
Sep 08 15:21:19.268197 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.268181    2568 scope.go:117] "RemoveContainer" containerID="d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"
Sep 08 15:21:19.268469 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:21:19.268452    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da\": container with ID starting with d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da not found: ID does not exist" containerID="d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"
Sep 08 15:21:19.268525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.268477    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da"} err="failed to get container status \"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da\": rpc error: code = NotFound desc = could not find container \"d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da\": container with ID starting with d7795c1ab3d5d2530e4bf28432ac47b3a1c5a7f034033043e93d18c8867489da not found: ID does not exist"
Sep 08 15:21:19.274210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.274189    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:21:19.277871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:19.277853    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-cqvlt"]
Sep 08 15:21:20.675059 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:21:20.675026    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e81d7a35-3920-4846-9fc5-f9fd842b0075" path="/var/lib/kubelet/pods/e81d7a35-3920-4846-9fc5-f9fd842b0075/volumes"
Sep 08 15:22:01.646148 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:01.646114    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:22:24.706633 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:24.706598    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:22:28.996730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:28.996698    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:22:38.602783 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:38.602746    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:22:38.603597 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:38.603572    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:22:44.803196 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:44.803158    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:22:52.714561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.714526    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"]
Sep 08 15:22:52.715001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.714830    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e81d7a35-3920-4846-9fc5-f9fd842b0075" containerName="authorino"
Sep 08 15:22:52.715001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.714841    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e81d7a35-3920-4846-9fc5-f9fd842b0075" containerName="authorino"
Sep 08 15:22:52.715001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.714893    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e81d7a35-3920-4846-9fc5-f9fd842b0075" containerName="authorino"
Sep 08 15:22:52.722533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.722484    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.725749 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.725722    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 15:22:52.725909 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.725783    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 15:22:52.725909 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.725834    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8qwbs\""
Sep 08 15:22:52.725909 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.725789    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 15:22:52.728020 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.727996    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"]
Sep 08 15:22:52.777256 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777222    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.777460 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777286    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2bs\" (UniqueName: \"kubernetes.io/projected/d5829140-031c-4f1c-8406-1108063e1a34-kube-api-access-fw2bs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.777460 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777371    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.777460 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777447    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5829140-031c-4f1c-8406-1108063e1a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.777590 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777489    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.777590 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.777515    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878349 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878315    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2bs\" (UniqueName: \"kubernetes.io/projected/d5829140-031c-4f1c-8406-1108063e1a34-kube-api-access-fw2bs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878349 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5829140-031c-4f1c-8406-1108063e1a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878408    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878425    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878450    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878767 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878733    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878788    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.878912 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.878883    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.880734 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.880715    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5829140-031c-4f1c-8406-1108063e1a34-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.881045 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.881025    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5829140-031c-4f1c-8406-1108063e1a34-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:52.887913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:52.887891    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2bs\" (UniqueName: \"kubernetes.io/projected/d5829140-031c-4f1c-8406-1108063e1a34-kube-api-access-fw2bs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p7kgf\" (UID: \"d5829140-031c-4f1c-8406-1108063e1a34\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:53.034773 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:53.034682    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:22:53.308937 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:53.308913    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"]
Sep 08 15:22:53.310972 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:22:53.310944    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5829140_031c_4f1c_8406_1108063e1a34.slice/crio-a75cf7fe5be86146c30ed53a3e93234dfea903f677cbc70664bce7a5b353acef WatchSource:0}: Error finding container a75cf7fe5be86146c30ed53a3e93234dfea903f677cbc70664bce7a5b353acef: Status 404 returned error can't find the container with id a75cf7fe5be86146c30ed53a3e93234dfea903f677cbc70664bce7a5b353acef
Sep 08 15:22:53.312364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:53.312348    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:22:53.733556 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:53.733508    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf" event={"ID":"d5829140-031c-4f1c-8406-1108063e1a34","Type":"ContainerStarted","Data":"a75cf7fe5be86146c30ed53a3e93234dfea903f677cbc70664bce7a5b353acef"}
Sep 08 15:22:54.095920 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:54.095825    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:22:54.739736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:54.739698    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf" event={"ID":"d5829140-031c-4f1c-8406-1108063e1a34","Type":"ContainerStarted","Data":"cdb130e9b96c49d2ffa1d8c4daaa5b1d8e3f4b87769947a4c67af798e2a6df68"}
Sep 08 15:22:54.780791 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:22:54.780740    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf" podStartSLOduration=1.762732948 podStartE2EDuration="2.780724194s" totalImagesPullingTime="1.017991246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:22:54.779148527 +0000 UTC m=+616.621945773" watchObservedRunningTime="2026-09-08 15:22:54.780724194 +0000 UTC m=+616.623521417"
Sep 08 15:23:00.096132 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:00.096099    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:23:03.035577 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:03.035501    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:23:03.035577 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:03.035560    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:23:03.048083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:03.048052    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:23:03.795115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:03.795082    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p7kgf"
Sep 08 15:23:15.600736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:15.600698    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:23:23.401786 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.401756    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"]
Sep 08 15:23:23.405374 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.405351    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.407815 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.407795    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 15:23:23.414885 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.414863    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"]
Sep 08 15:23:23.536743 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536707    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.536940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536768    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.536940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536795    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.536940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536823    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.536940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536838    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjwhg\" (UniqueName: \"kubernetes.io/projected/d8e477ae-9b79-4fc9-a857-8ba500f885ba-kube-api-access-pjwhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.536940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.536915    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638384 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638389    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjwhg\" (UniqueName: \"kubernetes.io/projected/d8e477ae-9b79-4fc9-a857-8ba500f885ba-kube-api-access-pjwhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638459    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638513    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638538    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638897 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638875    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638952 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638915    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.638996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.638960    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.640675 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.640657    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8e477ae-9b79-4fc9-a857-8ba500f885ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.641004 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.640989    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e477ae-9b79-4fc9-a857-8ba500f885ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.646746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.646724    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjwhg\" (UniqueName: \"kubernetes.io/projected/d8e477ae-9b79-4fc9-a857-8ba500f885ba-kube-api-access-pjwhg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s\" (UID: \"d8e477ae-9b79-4fc9-a857-8ba500f885ba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.714726 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.714648    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:23.986730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:23.986703    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"]
Sep 08 15:23:23.988650 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:23:23.988621    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8e477ae_9b79_4fc9_a857_8ba500f885ba.slice/crio-44758b945110ffa1ee593f6d71f27a5a4676d20984f6b770d082ceb557b002b2 WatchSource:0}: Error finding container 44758b945110ffa1ee593f6d71f27a5a4676d20984f6b770d082ceb557b002b2: Status 404 returned error can't find the container with id 44758b945110ffa1ee593f6d71f27a5a4676d20984f6b770d082ceb557b002b2
Sep 08 15:23:24.887576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:24.887545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s" event={"ID":"d8e477ae-9b79-4fc9-a857-8ba500f885ba","Type":"ContainerStarted","Data":"582b416b35512f48b8caa3523a64cc1eb0e4558b92bb5c8f43c4abed1efbbebc"}
Sep 08 15:23:24.887576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:24.887581    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s" event={"ID":"d8e477ae-9b79-4fc9-a857-8ba500f885ba","Type":"ContainerStarted","Data":"44758b945110ffa1ee593f6d71f27a5a4676d20984f6b770d082ceb557b002b2"}
Sep 08 15:23:24.906089 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:24.906041    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s" podStartSLOduration=1.64770694 podStartE2EDuration="1.906025703s" totalImagesPullingTime="258.318763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:23:24.904112887 +0000 UTC m=+646.746910109" watchObservedRunningTime="2026-09-08 15:23:24.906025703 +0000 UTC m=+646.748822926"
Sep 08 15:23:26.002796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:26.002763    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:23:33.715398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:33.715360    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:33.715822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:33.715434    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:33.727894 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:33.727869    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:23:33.943876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:23:33.943840    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s"
Sep 08 15:24:36.406315 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.406220    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d95999f5c-g47pq"]
Sep 08 15:24:36.409622 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.409604    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.412255 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.412234    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 15:24:36.412386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.412235    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bm7bp\""
Sep 08 15:24:36.416667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.416641    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d95999f5c-g47pq"]
Sep 08 15:24:36.513274 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.513239    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5729da77-8fd7-4305-9730-ca1e4107c4fd-tls-cert\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.513479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.513280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr749\" (UniqueName: \"kubernetes.io/projected/5729da77-8fd7-4305-9730-ca1e4107c4fd-kube-api-access-xr749\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.614512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.614467    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5729da77-8fd7-4305-9730-ca1e4107c4fd-tls-cert\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.614512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.614515    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr749\" (UniqueName: \"kubernetes.io/projected/5729da77-8fd7-4305-9730-ca1e4107c4fd-kube-api-access-xr749\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.617095 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.617070    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5729da77-8fd7-4305-9730-ca1e4107c4fd-tls-cert\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.622736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.622711    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr749\" (UniqueName: \"kubernetes.io/projected/5729da77-8fd7-4305-9730-ca1e4107c4fd-kube-api-access-xr749\") pod \"authorino-7d95999f5c-g47pq\" (UID: \"5729da77-8fd7-4305-9730-ca1e4107c4fd\") " pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.720887 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.720798    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d95999f5c-g47pq"
Sep 08 15:24:36.993127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:36.993051    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d95999f5c-g47pq"]
Sep 08 15:24:36.997658 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:24:36.996667    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5729da77_8fd7_4305_9730_ca1e4107c4fd.slice/crio-04c836a9c8431b742127e470893a0d82193df2ff913f16620a764331c1d09623 WatchSource:0}: Error finding container 04c836a9c8431b742127e470893a0d82193df2ff913f16620a764331c1d09623: Status 404 returned error can't find the container with id 04c836a9c8431b742127e470893a0d82193df2ff913f16620a764331c1d09623
Sep 08 15:24:37.258961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:37.258871    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d95999f5c-g47pq" event={"ID":"5729da77-8fd7-4305-9730-ca1e4107c4fd","Type":"ContainerStarted","Data":"04c836a9c8431b742127e470893a0d82193df2ff913f16620a764331c1d09623"}
Sep 08 15:24:38.267547 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:38.267512    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d95999f5c-g47pq" event={"ID":"5729da77-8fd7-4305-9730-ca1e4107c4fd","Type":"ContainerStarted","Data":"4b3937058bc20cdbb82b175f29e36f5832627c95349307f095e8bb7f0c40c39f"}
Sep 08 15:24:38.283389 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:38.283327    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d95999f5c-g47pq" podStartSLOduration=1.237925667 podStartE2EDuration="2.2833061s" totalImagesPullingTime="1.045380433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:24:38.281778093 +0000 UTC m=+720.124575339" watchObservedRunningTime="2026-09-08 15:24:38.2833061 +0000 UTC m=+720.126103319"
Sep 08 15:24:57.471041 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.471006    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:24:57.474897 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.474880    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:24:57.480438 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.480418    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8vmw2\""
Sep 08 15:24:57.488729 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.488704    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:24:57.490439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.490415    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5gvd\" (UniqueName: \"kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd\") pod \"test-curl-7778-139922421788080\" (UID: \"fd91d210-c35f-4a5c-b093-2aecbb242fa4\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:24:57.590900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.590864    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5gvd\" (UniqueName: \"kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd\") pod \"test-curl-7778-139922421788080\" (UID: \"fd91d210-c35f-4a5c-b093-2aecbb242fa4\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:24:57.600581 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.600557    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5gvd\" (UniqueName: \"kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd\") pod \"test-curl-7778-139922421788080\" (UID: \"fd91d210-c35f-4a5c-b093-2aecbb242fa4\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:24:57.783913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:57.783832    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:24:58.064472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:58.064445    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:24:58.066705 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:24:58.066642    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd91d210_c35f_4a5c_b093_2aecbb242fa4.slice/crio-a5a8790eea6115057ca24d3522e4fe7cc6756561cc45966c72187ed354e83352 WatchSource:0}: Error finding container a5a8790eea6115057ca24d3522e4fe7cc6756561cc45966c72187ed354e83352: Status 404 returned error can't find the container with id a5a8790eea6115057ca24d3522e4fe7cc6756561cc45966c72187ed354e83352
Sep 08 15:24:58.380277 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:58.380246    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139922421788080" event={"ID":"fd91d210-c35f-4a5c-b093-2aecbb242fa4","Type":"ContainerStarted","Data":"a5a8790eea6115057ca24d3522e4fe7cc6756561cc45966c72187ed354e83352"}
Sep 08 15:24:59.402462 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:24:59.402420    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:25:02.403802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:02.403770    2568 generic.go:356] "Generic (PLEG): container finished" podID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" containerID="3a2c83991ac65e702e31f69402bd4cca2c9efe23a9cb73a32512e6ea56479a30" exitCode=0
Sep 08 15:25:02.404169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:02.403809    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139922421788080" event={"ID":"fd91d210-c35f-4a5c-b093-2aecbb242fa4","Type":"ContainerDied","Data":"3a2c83991ac65e702e31f69402bd4cca2c9efe23a9cb73a32512e6ea56479a30"}
Sep 08 15:25:03.543320 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.543265    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:03.579242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.579215    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139922421788080_fd91d210-c35f-4a5c-b093-2aecbb242fa4/test-curl-7778-139922421788080/0.log"
Sep 08 15:25:03.604639 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.604607    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:03.607732 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.607700    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:03.640058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.640029    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5gvd\" (UniqueName: \"kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd\") pod \"fd91d210-c35f-4a5c-b093-2aecbb242fa4\" (UID: \"fd91d210-c35f-4a5c-b093-2aecbb242fa4\") "
Sep 08 15:25:03.642338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.642308    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd" (OuterVolumeSpecName: "kube-api-access-s5gvd") pod "fd91d210-c35f-4a5c-b093-2aecbb242fa4" (UID: "fd91d210-c35f-4a5c-b093-2aecbb242fa4"). InnerVolumeSpecName "kube-api-access-s5gvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:25:03.727468 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.727376    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:03.727732 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.727713    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:03.727732 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.727728    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:03.727900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.727795    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:03.730658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.730638    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:03.738339 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.738314    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:03.740664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.740630    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2rk\" (UniqueName: \"kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk\") pod \"test-curl-7778-139922421788080\" (UID: \"0455065a-541f-4d3c-82b5-ba8189bfd52e\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:03.741260 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.741239    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5gvd\" (UniqueName: \"kubernetes.io/projected/fd91d210-c35f-4a5c-b093-2aecbb242fa4-kube-api-access-s5gvd\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:25:03.842054 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.842014    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2rk\" (UniqueName: \"kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk\") pod \"test-curl-7778-139922421788080\" (UID: \"0455065a-541f-4d3c-82b5-ba8189bfd52e\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:03.850149 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:03.850123    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2rk\" (UniqueName: \"kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk\") pod \"test-curl-7778-139922421788080\" (UID: \"0455065a-541f-4d3c-82b5-ba8189bfd52e\") " pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:04.041417 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.041287    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:04.313096 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.313068    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:04.315084 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:04.315055    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0455065a_541f_4d3c_82b5_ba8189bfd52e.slice/crio-91d88e350a81515e3558abe773d0fab629e560922e3beaaca18d938fabda4925 WatchSource:0}: Error finding container 91d88e350a81515e3558abe773d0fab629e560922e3beaaca18d938fabda4925: Status 404 returned error can't find the container with id 91d88e350a81515e3558abe773d0fab629e560922e3beaaca18d938fabda4925
Sep 08 15:25:04.416224 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.416192    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:04.416423 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.416231    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5a8790eea6115057ca24d3522e4fe7cc6756561cc45966c72187ed354e83352"
Sep 08 15:25:04.417659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.417633    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139922421788080" event={"ID":"0455065a-541f-4d3c-82b5-ba8189bfd52e","Type":"ContainerStarted","Data":"91d88e350a81515e3558abe773d0fab629e560922e3beaaca18d938fabda4925"}
Sep 08 15:25:04.431284 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.431250    2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7778-139922421788080" oldPodUID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e"
Sep 08 15:25:04.675453 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:04.675418    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd91d210-c35f-4a5c-b093-2aecbb242fa4" path="/var/lib/kubelet/pods/fd91d210-c35f-4a5c-b093-2aecbb242fa4/volumes"
Sep 08 15:25:05.426205 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:05.426158    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139922421788080" event={"ID":"0455065a-541f-4d3c-82b5-ba8189bfd52e","Type":"ContainerStarted","Data":"4f35679f993da399c3f8c62fe87982b2199d65246e277137ec198c2c55b093ff"}
Sep 08 15:25:05.440883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:05.440834    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7778-139922421788080" podStartSLOduration=1.422320776 podStartE2EDuration="2.440817183s" totalImagesPullingTime="1.018496407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:05.439649297 +0000 UTC m=+747.282446520" watchObservedRunningTime="2026-09-08 15:25:05.440817183 +0000 UTC m=+747.283614406"
Sep 08 15:25:05.534480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:05.534443    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7778-139922421788080_0455065a-541f-4d3c-82b5-ba8189bfd52e/test-curl-7778-139922421788080/0.log"
Sep 08 15:25:06.433000 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:06.432962    2568 generic.go:356] "Generic (PLEG): container finished" podID="0455065a-541f-4d3c-82b5-ba8189bfd52e" containerID="4f35679f993da399c3f8c62fe87982b2199d65246e277137ec198c2c55b093ff" exitCode=0
Sep 08 15:25:06.433412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:06.433041    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7778-139922421788080" event={"ID":"0455065a-541f-4d3c-82b5-ba8189bfd52e","Type":"ContainerDied","Data":"4f35679f993da399c3f8c62fe87982b2199d65246e277137ec198c2c55b093ff"}
Sep 08 15:25:07.565199 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.565176    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:07.601567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.601535    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:07.605149 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.605122    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7778-139922421788080"]
Sep 08 15:25:07.674676 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.674642    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jt2rk\" (UniqueName: \"kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk\") pod \"0455065a-541f-4d3c-82b5-ba8189bfd52e\" (UID: \"0455065a-541f-4d3c-82b5-ba8189bfd52e\") "
Sep 08 15:25:07.676657 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.676629    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk" (OuterVolumeSpecName: "kube-api-access-jt2rk") pod "0455065a-541f-4d3c-82b5-ba8189bfd52e" (UID: "0455065a-541f-4d3c-82b5-ba8189bfd52e"). InnerVolumeSpecName "kube-api-access-jt2rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:25:07.775904 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:07.775816    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2rk\" (UniqueName: \"kubernetes.io/projected/0455065a-541f-4d3c-82b5-ba8189bfd52e-kube-api-access-jt2rk\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:25:08.445811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:08.445782    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:08.445811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:08.445813    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91d88e350a81515e3558abe773d0fab629e560922e3beaaca18d938fabda4925"
Sep 08 15:25:08.455537 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:08.455507    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139922421788080\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:08.673728 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:08.673686    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7778-139922421788080\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" pod="opendatahub/test-curl-7778-139922421788080"
Sep 08 15:25:08.674803 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:08.674781    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" path="/var/lib/kubelet/pods/0455065a-541f-4d3c-82b5-ba8189bfd52e/volumes"
Sep 08 15:25:12.535627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.535593    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:25:12.536076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.535910    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:12.536076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.535921    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:12.536076 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.535987    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0455065a-541f-4d3c-82b5-ba8189bfd52e" containerName="test-curl-7778-139922421788080"
Sep 08 15:25:12.539249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.539223    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.541658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.541634    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:25:12.541791 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.541699    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:25:12.542547 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.542525    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-98d03a-serving-cert\""
Sep 08 15:25:12.542681 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.542569    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:25:12.542800 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.542783    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4m2fn\""
Sep 08 15:25:12.549547 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.549525    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:25:12.619143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.619112    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.619282 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.619156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.619282 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.619213    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.619282 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.619234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.719867 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.719841    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.720034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.719880    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.720034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.719918    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.720034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.719938    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.720204 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:12.720036    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-serving-cert: secret "maas-api-e2e-autores-98d03a-serving-cert" not found
Sep 08 15:25:12.720204 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:12.720104    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls podName:c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6 nodeName:}" failed. No retries permitted until 2026-09-08 15:25:13.22008513 +0000 UTC m=+755.062882348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls") pod "maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6") : secret "maas-api-e2e-autores-98d03a-serving-cert" not found
Sep 08 15:25:12.720671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.720645    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.722308 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.722268    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:12.747434 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:12.747408    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:13.225078 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:13.225041    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:13.225266 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:13.225191    2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-serving-cert: secret "maas-api-e2e-autores-98d03a-serving-cert" not found
Sep 08 15:25:13.225266 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:25:13.225251    2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls podName:c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6 nodeName:}" failed. No retries permitted until 2026-09-08 15:25:14.225237093 +0000 UTC m=+756.068034294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls") pod "maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6") : secret "maas-api-e2e-autores-98d03a-serving-cert" not found
Sep 08 15:25:13.807977 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:13.807944    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:25:14.235267 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:14.235233    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:14.237630 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:14.237607    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:14.354150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:14.354119    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:14.632724 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:14.632701    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:25:14.634883 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:14.634854    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89eb639_ac9c_4c62_b3e0_c01e66f9ffe6.slice/crio-896f9b71a4d1e044ca321ca79d711ca8b70a395d4146d5efa12aac4b5d72d118 WatchSource:0}: Error finding container 896f9b71a4d1e044ca321ca79d711ca8b70a395d4146d5efa12aac4b5d72d118: Status 404 returned error can't find the container with id 896f9b71a4d1e044ca321ca79d711ca8b70a395d4146d5efa12aac4b5d72d118
Sep 08 15:25:15.484872 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:15.484835    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" event={"ID":"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6","Type":"ContainerStarted","Data":"896f9b71a4d1e044ca321ca79d711ca8b70a395d4146d5efa12aac4b5d72d118"}
Sep 08 15:25:16.491800 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:16.491714    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" event={"ID":"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6","Type":"ContainerStarted","Data":"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"}
Sep 08 15:25:16.492184 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:16.491868    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:16.509112 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:16.509061    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" podStartSLOduration=2.9134516599999998 podStartE2EDuration="4.50904668s" totalImagesPullingTime="1.59559502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:16.507574711 +0000 UTC m=+758.350371937" watchObservedRunningTime="2026-09-08 15:25:16.50904668 +0000 UTC m=+758.351843894"
Sep 08 15:25:22.502497 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:22.502465    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:25:28.875399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.875356    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:25:28.883147 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.883113    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:28.886127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.886105    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b75273-serving-cert\""
Sep 08 15:25:28.889571 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.889544    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:25:28.969185 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.969156    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:28.969185 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.969188    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:28.969449 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.969214    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlpz\" (UniqueName: \"kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:28.969449 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:28.969234    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.069848 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.069820    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlpz\" (UniqueName: \"kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.069848 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.069855    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.070049 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.069915    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.070049 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.069935    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.070575 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.070550    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.072309 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.072265    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.072506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.072487    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.078028 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.078011    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlpz\" (UniqueName: \"kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz\") pod \"maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.196205 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.196120    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:29.470121 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.470090    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:25:29.471037 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:29.471011    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7828e655_c838_442e_a454_fe670d8c45dc.slice/crio-f52b14dc5dfa1ce620b29ff2ddf88ddcdcd1ae2279bcbd7483ad912d2ba5e558 WatchSource:0}: Error finding container f52b14dc5dfa1ce620b29ff2ddf88ddcdcd1ae2279bcbd7483ad912d2ba5e558: Status 404 returned error can't find the container with id f52b14dc5dfa1ce620b29ff2ddf88ddcdcd1ae2279bcbd7483ad912d2ba5e558
Sep 08 15:25:29.566451 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.566419    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" event={"ID":"7828e655-c838-442e-a454-fe670d8c45dc","Type":"ContainerStarted","Data":"f52b14dc5dfa1ce620b29ff2ddf88ddcdcd1ae2279bcbd7483ad912d2ba5e558"}
Sep 08 15:25:29.686609 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.686582    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:25:29.691060 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.691041    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.693355 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.693335    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b75273-dockercfg-x5dwd\""
Sep 08 15:25:29.693457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.693426    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b75273\""
Sep 08 15:25:29.699209 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.699183    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:25:29.776119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.776040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.776119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.776080    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnjk\" (UniqueName: \"kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.876789 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.876761    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.877085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.876796    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnjk\" (UniqueName: \"kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.877464 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.877432    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:29.884334 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:29.884313    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnjk\" (UniqueName: \"kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk\") pod \"payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:30.002872 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.002836    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:30.273353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.273318    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:25:30.276792 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:30.276767    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5214ae18_243f_461f_bc2f_e118a8e46242.slice/crio-6c51cd1477a5b1c642bbc2bc16b883dc9d8455899a20f292115d8c34cb258080 WatchSource:0}: Error finding container 6c51cd1477a5b1c642bbc2bc16b883dc9d8455899a20f292115d8c34cb258080: Status 404 returned error can't find the container with id 6c51cd1477a5b1c642bbc2bc16b883dc9d8455899a20f292115d8c34cb258080
Sep 08 15:25:30.573790 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.573751    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" event={"ID":"7828e655-c838-442e-a454-fe670d8c45dc","Type":"ContainerStarted","Data":"75ac0fcf22509adc975fb676c36a8e8516029202381e1291add84825071fc7fb"}
Sep 08 15:25:30.573987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.573906    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:30.575101 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.575078    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" event={"ID":"5214ae18-243f-461f-bc2f-e118a8e46242","Type":"ContainerStarted","Data":"6c51cd1477a5b1c642bbc2bc16b883dc9d8455899a20f292115d8c34cb258080"}
Sep 08 15:25:30.592845 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:30.592803    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" podStartSLOduration=2.219428885 podStartE2EDuration="2.592789175s" totalImagesPullingTime="373.36029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:30.59094459 +0000 UTC m=+772.433741829" watchObservedRunningTime="2026-09-08 15:25:30.592789175 +0000 UTC m=+772.435586411"
Sep 08 15:25:31.830429 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:31.830393    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:25:32.587217 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:32.587180    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" event={"ID":"5214ae18-243f-461f-bc2f-e118a8e46242","Type":"ContainerStarted","Data":"d167c2cee1793e3fae6fbf2538d5dad5cf4d2a42f9cbd2204ff7d964bfec26aa"}
Sep 08 15:25:32.587399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:32.587255    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:32.603166 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:32.603121    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" podStartSLOduration=1.768263127 podStartE2EDuration="3.603106133s" totalImagesPullingTime="1.834843006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:32.601272764 +0000 UTC m=+774.444069988" watchObservedRunningTime="2026-09-08 15:25:32.603106133 +0000 UTC m=+774.445903411"
Sep 08 15:25:36.586746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:36.586718    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:25:38.802345 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.802314    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:25:38.805561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.805543    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.807615 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.807596    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-d4257732-kserve-self-signed-certs\""
Sep 08 15:25:38.815903 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.815872    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:25:38.955392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955355    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.955574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955411    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.955574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955488    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.955574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955524    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ch9\" (UniqueName: \"kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.955574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:38.955790 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:38.955649    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.056756 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056671    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.056925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056794    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.056925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056825    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.056925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056846    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.056925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056876    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ch9\" (UniqueName: \"kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.057164 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.056930    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.057164 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.057111    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.057271 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.057246    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.057356 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.057272    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.059073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.059050    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.059367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.059349    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.069427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.065241    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ch9\" (UniqueName: \"kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9\") pod \"e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") " pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.114269 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.114233    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:39.398596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.398568    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:25:39.400331 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:39.400273    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6cb2376_48b8_4fa8_9559_239e2c40bb0f.slice/crio-0c51fcfb36f9244267eba243a90ac5041c1efefd66c357c7e090046359a4a097 WatchSource:0}: Error finding container 0c51fcfb36f9244267eba243a90ac5041c1efefd66c357c7e090046359a4a097: Status 404 returned error can't find the container with id 0c51fcfb36f9244267eba243a90ac5041c1efefd66c357c7e090046359a4a097
Sep 08 15:25:39.627346 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.627308    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" event={"ID":"d6cb2376-48b8-4fa8-9559-239e2c40bb0f","Type":"ContainerStarted","Data":"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"}
Sep 08 15:25:39.627527 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.627362    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" event={"ID":"d6cb2376-48b8-4fa8-9559-239e2c40bb0f","Type":"ContainerStarted","Data":"0c51fcfb36f9244267eba243a90ac5041c1efefd66c357c7e090046359a4a097"}
Sep 08 15:25:39.644622 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:39.644576    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" podStartSLOduration=1.644562659 podStartE2EDuration="1.644562659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:39.642069662 +0000 UTC m=+781.484866880" watchObservedRunningTime="2026-09-08 15:25:39.644562659 +0000 UTC m=+781.487359902"
Sep 08 15:25:43.594679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:43.594645    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:25:46.701110 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:46.701073    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:25:49.114975 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:49.114944    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:49.115447 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:49.115217    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:49.129285 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:49.129259    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:49.691242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:49.691212    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:25:53.514899 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.514856    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:25:53.519137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.519117    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.521901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.521878    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c69c6364-serving-cert\""
Sep 08 15:25:53.531058 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.531033    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:25:53.579510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.579479    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.579666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.579533    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.579666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.579550    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rvx2\" (UniqueName: \"kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.579666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.579587    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.680114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.680081    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.680268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.680146    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.680268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.680181    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.680268 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.680197    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rvx2\" (UniqueName: \"kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.680843 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.680811    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.682654 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.682635    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.682741 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.682699    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.688452 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.688429    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rvx2\" (UniqueName: \"kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2\") pod \"maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:53.834857 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:53.834786    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:54.106969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:54.106941    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:25:54.108692 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:25:54.108665    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca369ee3_ad77_4829_9a06_7ea425d3dd4b.slice/crio-70e506124aef8af7e67bee7418dadcb2dec6acae93d034c0a3b87085c3a0058f WatchSource:0}: Error finding container 70e506124aef8af7e67bee7418dadcb2dec6acae93d034c0a3b87085c3a0058f: Status 404 returned error can't find the container with id 70e506124aef8af7e67bee7418dadcb2dec6acae93d034c0a3b87085c3a0058f
Sep 08 15:25:54.706606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:54.706569    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" event={"ID":"ca369ee3-ad77-4829-9a06-7ea425d3dd4b","Type":"ContainerStarted","Data":"70e506124aef8af7e67bee7418dadcb2dec6acae93d034c0a3b87085c3a0058f"}
Sep 08 15:25:55.713970 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:55.713931    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" event={"ID":"ca369ee3-ad77-4829-9a06-7ea425d3dd4b","Type":"ContainerStarted","Data":"46e603c9a0496a03d28b4f4fba970e5940fe0ef14e92890fc231ed2f9afc429e"}
Sep 08 15:25:55.714345 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:55.714061    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:25:55.733413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:25:55.733362    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" podStartSLOduration=1.503837547 podStartE2EDuration="2.733349644s" totalImagesPullingTime="1.229512097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:55.730528252 +0000 UTC m=+797.573325474" watchObservedRunningTime="2026-09-08 15:25:55.733349644 +0000 UTC m=+797.576146867"
Sep 08 15:26:01.725044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:01.724974    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:26:03.799977 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:03.799939    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:26:10.024253 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.024219    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:26:10.028766 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.028747    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.030910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.030886    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-98d03a\"/\"kube-root-ca.crt\""
Sep 08 15:26:10.031806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.031782    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-98d03a\"/\"openshift-service-ca.crt\""
Sep 08 15:26:10.031930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.031854    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-98d03a\"/\"autores-model-98d03a-kserve-self-signed-certs\""
Sep 08 15:26:10.031930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.031859    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-98d03a\"/\"default-dockercfg-frnr9\""
Sep 08 15:26:10.036374 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.036350    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:26:10.129557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129525    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.129705 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129567    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.129750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7255j\" (UniqueName: \"kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.129750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129734    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.129817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129799    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.129856 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.129832    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231173    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7255j\" (UniqueName: \"kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231208    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231243    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231273    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231318    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231351    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231675 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231631    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231769 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.231810 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.231756    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.233454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.233435    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.233707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.233690    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.239383 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.239360    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7255j\" (UniqueName: \"kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j\") pod \"autores-model-98d03a-kserve-5b45d5486c-cx8s8\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") " pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.339549 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.339482    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:10.615529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.615501    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:26:10.616688 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:26:10.616664    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2414c2e_8f1f_457e_8dd2_ba016b9fe7eb.slice/crio-0cfed83cb48f815e96bcbb25a33e7dfd687932951f3908d05e256ad3603e3a0f WatchSource:0}: Error finding container 0cfed83cb48f815e96bcbb25a33e7dfd687932951f3908d05e256ad3603e3a0f: Status 404 returned error can't find the container with id 0cfed83cb48f815e96bcbb25a33e7dfd687932951f3908d05e256ad3603e3a0f
Sep 08 15:26:10.796518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.796480    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" event={"ID":"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb","Type":"ContainerStarted","Data":"63af077a7e3538d8bdcaedf173c275ccdda741185b164c69f3a46f697a07dc92"}
Sep 08 15:26:10.796674 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.796526    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" event={"ID":"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb","Type":"ContainerStarted","Data":"0cfed83cb48f815e96bcbb25a33e7dfd687932951f3908d05e256ad3603e3a0f"}
Sep 08 15:26:10.830528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:10.830484    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" podStartSLOduration=0.830469844 podStartE2EDuration="830.469844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:10.827861009 +0000 UTC m=+812.670658242" watchObservedRunningTime="2026-09-08 15:26:10.830469844 +0000 UTC m=+812.673267066"
Sep 08 15:26:17.003584 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:17.003548    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:26:20.339591 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:20.339556    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:20.339980 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:20.339607    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:20.352141 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:20.352118    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:20.863936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:20.863906    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:26:24.754217 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:24.754184    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:26:24.754706 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:24.754443    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" podUID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" containerName="maas-api" containerID="cri-o://46e603c9a0496a03d28b4f4fba970e5940fe0ef14e92890fc231ed2f9afc429e" gracePeriod=30
Sep 08 15:26:24.877243 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:24.877214    2568 generic.go:356] "Generic (PLEG): container finished" podID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" containerID="46e603c9a0496a03d28b4f4fba970e5940fe0ef14e92890fc231ed2f9afc429e" exitCode=0
Sep 08 15:26:24.877399 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:24.877277    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" event={"ID":"ca369ee3-ad77-4829-9a06-7ea425d3dd4b","Type":"ContainerDied","Data":"46e603c9a0496a03d28b4f4fba970e5940fe0ef14e92890fc231ed2f9afc429e"}
Sep 08 15:26:25.000187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.000163    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:26:25.163546 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.163511    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rvx2\" (UniqueName: \"kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2\") pod \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") "
Sep 08 15:26:25.163722 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.163560    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca\") pod \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") "
Sep 08 15:26:25.163722 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.163582    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs\") pod \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") "
Sep 08 15:26:25.163722 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.163605    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls\") pod \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\" (UID: \"ca369ee3-ad77-4829-9a06-7ea425d3dd4b\") "
Sep 08 15:26:25.164100 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.164065    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca369ee3-ad77-4829-9a06-7ea425d3dd4b" (UID: "ca369ee3-ad77-4829-9a06-7ea425d3dd4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:26:25.165878 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.165850    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca369ee3-ad77-4829-9a06-7ea425d3dd4b" (UID: "ca369ee3-ad77-4829-9a06-7ea425d3dd4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:25.165970 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.165900    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca369ee3-ad77-4829-9a06-7ea425d3dd4b" (UID: "ca369ee3-ad77-4829-9a06-7ea425d3dd4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:25.165970 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.165914    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2" (OuterVolumeSpecName: "kube-api-access-7rvx2") pod "ca369ee3-ad77-4829-9a06-7ea425d3dd4b" (UID: "ca369ee3-ad77-4829-9a06-7ea425d3dd4b"). InnerVolumeSpecName "kube-api-access-7rvx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:26:25.264454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.264419    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rvx2\" (UniqueName: \"kubernetes.io/projected/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-kube-api-access-7rvx2\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:25.264454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.264446    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:25.264454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.264457    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:25.264677 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.264465    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca369ee3-ad77-4829-9a06-7ea425d3dd4b-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:25.884084 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.884048    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm" event={"ID":"ca369ee3-ad77-4829-9a06-7ea425d3dd4b","Type":"ContainerDied","Data":"70e506124aef8af7e67bee7418dadcb2dec6acae93d034c0a3b87085c3a0058f"}
Sep 08 15:26:25.884084 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.884086    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"
Sep 08 15:26:25.884565 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.884121    2568 scope.go:117] "RemoveContainer" containerID="46e603c9a0496a03d28b4f4fba970e5940fe0ef14e92890fc231ed2f9afc429e"
Sep 08 15:26:25.912073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.912047    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:26:25.924016 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:25.923993    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c69c6364-797568b8db-6m5zm"]
Sep 08 15:26:26.676186 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:26.676151    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" path="/var/lib/kubelet/pods/ca369ee3-ad77-4829-9a06-7ea425d3dd4b/volumes"
Sep 08 15:26:29.632033 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.631995    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:26:29.632518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.632372    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" containerName="maas-api"
Sep 08 15:26:29.632518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.632384    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" containerName="maas-api"
Sep 08 15:26:29.632518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.632458    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca369ee3-ad77-4829-9a06-7ea425d3dd4b" containerName="maas-api"
Sep 08 15:26:29.644005 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.643977    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.646481 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.646456    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ecef8168-serving-cert\""
Sep 08 15:26:29.647412 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.647385    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:26:29.802634 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.802604    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.802807 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.802663    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmgg\" (UniqueName: \"kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.802807 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.802702    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.802807 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.802733    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.903652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.903561    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.903652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.903627    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmgg\" (UniqueName: \"kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.903888 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.903662    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.903888 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.903705    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.904485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.904458    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.906029 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.906008    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.906443 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.906425    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.911900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.911880    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmgg\" (UniqueName: \"kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg\") pod \"maas-api-e2e-ait-ecef8168-686588fcc9-crtxf\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:29.958146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:29.958119    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:30.232758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:30.232733    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:26:30.234287 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:26:30.234259    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603ee0a0_9063_44f5_81b2_f56298997403.slice/crio-2ea8edcaa3a83417ee57c5f34942821932406e95949950a9ef0b1efa8fce958d WatchSource:0}: Error finding container 2ea8edcaa3a83417ee57c5f34942821932406e95949950a9ef0b1efa8fce958d: Status 404 returned error can't find the container with id 2ea8edcaa3a83417ee57c5f34942821932406e95949950a9ef0b1efa8fce958d
Sep 08 15:26:30.914362 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:30.914327    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" event={"ID":"603ee0a0-9063-44f5-81b2-f56298997403","Type":"ContainerStarted","Data":"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"}
Sep 08 15:26:30.914763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:30.914381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" event={"ID":"603ee0a0-9063-44f5-81b2-f56298997403","Type":"ContainerStarted","Data":"2ea8edcaa3a83417ee57c5f34942821932406e95949950a9ef0b1efa8fce958d"}
Sep 08 15:26:30.914763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:30.914421    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:30.931880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:30.931830    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" podStartSLOduration=1.5638879700000001 podStartE2EDuration="1.931814061s" totalImagesPullingTime="367.926091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:30.928990156 +0000 UTC m=+832.771787370" watchObservedRunningTime="2026-09-08 15:26:30.931814061 +0000 UTC m=+832.774611299"
Sep 08 15:26:36.599127 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:36.599085    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:26:36.926051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:36.926024    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:26:52.345835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.345799    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:26:52.351249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.351229    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.353742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.353717    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-d4257732-kserve-self-signed-certs\""
Sep 08 15:26:52.355469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.355450    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:26:52.402325 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402280    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxlb8\" (UniqueName: \"kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.402431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402337    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.402431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402380    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.402431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402417    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.402543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.402543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.402466    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503021 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.502997    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxlb8\" (UniqueName: \"kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503029    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503044    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503082    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503110    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503142    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503476 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503448    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503509    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.503618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.503593    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.505321 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.505280    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.505602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.505584    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.510503 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.510482    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxlb8\" (UniqueName: \"kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8\") pod \"e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") " pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.662100 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.662062    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:26:52.931960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:52.931937    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:26:52.934274 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:26:52.934246    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57c1ae89_ef9a_4f4b_a32b_a3f89fe06b4b.slice/crio-ecb437a19f54f22e264ec5de535f89e204f0ffb0d9ec477c7529785a5c68ac3f WatchSource:0}: Error finding container ecb437a19f54f22e264ec5de535f89e204f0ffb0d9ec477c7529785a5c68ac3f: Status 404 returned error can't find the container with id ecb437a19f54f22e264ec5de535f89e204f0ffb0d9ec477c7529785a5c68ac3f
Sep 08 15:26:53.038468 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:53.038430    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" event={"ID":"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b","Type":"ContainerStarted","Data":"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"}
Sep 08 15:26:53.038606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:53.038478    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" event={"ID":"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b","Type":"ContainerStarted","Data":"ecb437a19f54f22e264ec5de535f89e204f0ffb0d9ec477c7529785a5c68ac3f"}
Sep 08 15:26:53.055269 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:53.055219    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" podStartSLOduration=1.055202146 podStartE2EDuration="1.055202146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:53.053814229 +0000 UTC m=+854.896611453" watchObservedRunningTime="2026-09-08 15:26:53.055202146 +0000 UTC m=+854.897999369"
Sep 08 15:26:53.403930 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:53.403896    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:26:55.476319 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.476273    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:26:55.476754 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.476509    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" podUID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" containerName="maas-api" containerID="cri-o://893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4" gracePeriod=30
Sep 08 15:26:55.735839 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.735782    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:26:55.833965 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.833933    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") pod \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") "
Sep 08 15:26:55.834137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.833989    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca\") pod \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") "
Sep 08 15:26:55.834137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.834019    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf\") pod \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") "
Sep 08 15:26:55.834137 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.834067    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs\") pod \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\" (UID: \"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6\") "
Sep 08 15:26:55.834424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.834403    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:26:55.835908 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.835885    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:55.836069 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.836048    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:55.836222 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.836203    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf" (OuterVolumeSpecName: "kube-api-access-mk8rf") pod "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" (UID: "c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6"). InnerVolumeSpecName "kube-api-access-mk8rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:26:55.934811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.934779    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:55.934811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.934805    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:55.934811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.934815    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8rf\" (UniqueName: \"kubernetes.io/projected/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-kube-api-access-mk8rf\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:55.935019 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:55.934825    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:26:56.056430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.056338    2568 generic.go:356] "Generic (PLEG): container finished" podID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" containerID="893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4" exitCode=0
Sep 08 15:26:56.056430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.056375    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"
Sep 08 15:26:56.056613 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.056431    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" event={"ID":"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6","Type":"ContainerDied","Data":"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"}
Sep 08 15:26:56.056613 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.056484    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb" event={"ID":"c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6","Type":"ContainerDied","Data":"896f9b71a4d1e044ca321ca79d711ca8b70a395d4146d5efa12aac4b5d72d118"}
Sep 08 15:26:56.056613 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.056509    2568 scope.go:117] "RemoveContainer" containerID="893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"
Sep 08 15:26:56.066088 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.066074    2568 scope.go:117] "RemoveContainer" containerID="893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"
Sep 08 15:26:56.066368 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:26:56.066352    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4\": container with ID starting with 893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4 not found: ID does not exist" containerID="893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"
Sep 08 15:26:56.066420 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.066375    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4"} err="failed to get container status \"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4\": rpc error: code = NotFound desc = could not find container \"893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4\": container with ID starting with 893065184ee02677e0ef983a702f99f637e465938c90fd2f7fbf9f94706cb0c4 not found: ID does not exist"
Sep 08 15:26:56.076812 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.076790    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:26:56.081865 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.081847    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-98d03a-5cd6df44c-fgzdb"]
Sep 08 15:26:56.680762 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:26:56.680729    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" path="/var/lib/kubelet/pods/c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6/volumes"
Sep 08 15:27:00.163353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.163317    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:27:00.163844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.163548    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" podUID="603ee0a0-9063-44f5-81b2-f56298997403" containerName="maas-api" containerID="cri-o://ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84" gracePeriod=30
Sep 08 15:27:00.415696 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.415637    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:27:00.479556 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.479526    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bmgg\" (UniqueName: \"kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg\") pod \"603ee0a0-9063-44f5-81b2-f56298997403\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") "
Sep 08 15:27:00.479705 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.479610    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs\") pod \"603ee0a0-9063-44f5-81b2-f56298997403\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") "
Sep 08 15:27:00.479705 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.479664    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca\") pod \"603ee0a0-9063-44f5-81b2-f56298997403\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") "
Sep 08 15:27:00.479807 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.479784    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls\") pod \"603ee0a0-9063-44f5-81b2-f56298997403\" (UID: \"603ee0a0-9063-44f5-81b2-f56298997403\") "
Sep 08 15:27:00.480000 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.479976    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "603ee0a0-9063-44f5-81b2-f56298997403" (UID: "603ee0a0-9063-44f5-81b2-f56298997403"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:27:00.480133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.480106    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603ee0a0-9063-44f5-81b2-f56298997403-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:00.481728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.481699    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "603ee0a0-9063-44f5-81b2-f56298997403" (UID: "603ee0a0-9063-44f5-81b2-f56298997403"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:27:00.481835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.481755    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "603ee0a0-9063-44f5-81b2-f56298997403" (UID: "603ee0a0-9063-44f5-81b2-f56298997403"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:27:00.481835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.481760    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg" (OuterVolumeSpecName: "kube-api-access-6bmgg") pod "603ee0a0-9063-44f5-81b2-f56298997403" (UID: "603ee0a0-9063-44f5-81b2-f56298997403"). InnerVolumeSpecName "kube-api-access-6bmgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:27:00.581582 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.581549    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:00.581582 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.581577    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bmgg\" (UniqueName: \"kubernetes.io/projected/603ee0a0-9063-44f5-81b2-f56298997403-kube-api-access-6bmgg\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:00.581777 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:00.581590    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603ee0a0-9063-44f5-81b2-f56298997403-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:01.014546 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.014463    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:27:01.091398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.090313    2568 generic.go:356] "Generic (PLEG): container finished" podID="603ee0a0-9063-44f5-81b2-f56298997403" containerID="ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84" exitCode=0
Sep 08 15:27:01.091398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.090468    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" event={"ID":"603ee0a0-9063-44f5-81b2-f56298997403","Type":"ContainerDied","Data":"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"}
Sep 08 15:27:01.091398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.090547    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf" event={"ID":"603ee0a0-9063-44f5-81b2-f56298997403","Type":"ContainerDied","Data":"2ea8edcaa3a83417ee57c5f34942821932406e95949950a9ef0b1efa8fce958d"}
Sep 08 15:27:01.091398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.090588    2568 scope.go:117] "RemoveContainer" containerID="ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"
Sep 08 15:27:01.091398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.090790    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"
Sep 08 15:27:01.100381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.100276    2568 scope.go:117] "RemoveContainer" containerID="ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"
Sep 08 15:27:01.100577 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:27:01.100557    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84\": container with ID starting with ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84 not found: ID does not exist" containerID="ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"
Sep 08 15:27:01.100619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.100584    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84"} err="failed to get container status \"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84\": rpc error: code = NotFound desc = could not find container \"ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84\": container with ID starting with ace78e26487779ca039bf182a3402854340019d4a714f1f758bed511fd0fde84 not found: ID does not exist"
Sep 08 15:27:01.107035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.107008    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:27:01.110909 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:01.110888    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecef8168-686588fcc9-crtxf"]
Sep 08 15:27:02.662558 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:02.662517    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:02.662961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:02.662655    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:02.675881 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:02.675847    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603ee0a0-9063-44f5-81b2-f56298997403" path="/var/lib/kubelet/pods/603ee0a0-9063-44f5-81b2-f56298997403/volumes"
Sep 08 15:27:02.676234 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:02.676221    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:03.117405 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:03.117326    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:06.217551 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.217512    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:06.218039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218020    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603ee0a0-9063-44f5-81b2-f56298997403" containerName="maas-api"
Sep 08 15:27:06.218122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218043    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="603ee0a0-9063-44f5-81b2-f56298997403" containerName="maas-api"
Sep 08 15:27:06.218122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218091    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" containerName="maas-api"
Sep 08 15:27:06.218122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218100    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" containerName="maas-api"
Sep 08 15:27:06.218270 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218192    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89eb639-ac9c-4c62-b3e0-c01e66f9ffe6" containerName="maas-api"
Sep 08 15:27:06.218270 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.218211    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="603ee0a0-9063-44f5-81b2-f56298997403" containerName="maas-api"
Sep 08 15:27:06.221242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.221223    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.223371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.223354    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b5aeacc1-dockercfg-clhk7\""
Sep 08 15:27:06.223451 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.223407    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b5aeacc1\""
Sep 08 15:27:06.229409 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.229242    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:06.335370 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.335317    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd8c\" (UniqueName: \"kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.335538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.335398    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.436342 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.436285    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd8c\" (UniqueName: \"kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.436500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.436375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.437091 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.437069    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.444052 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.444026    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd8c\" (UniqueName: \"kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c\") pod \"payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.533150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.533076    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:06.806631 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:06.806608    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:06.808465 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:27:06.808438    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode652b03f_d42d_42d3_a9f0_1ccf80e0849a.slice/crio-1fdf8a089477846be9b2351a6487d3bea5398455864c3776e52231bde7ae76f8 WatchSource:0}: Error finding container 1fdf8a089477846be9b2351a6487d3bea5398455864c3776e52231bde7ae76f8: Status 404 returned error can't find the container with id 1fdf8a089477846be9b2351a6487d3bea5398455864c3776e52231bde7ae76f8
Sep 08 15:27:07.130750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:07.130718    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" event={"ID":"e652b03f-d42d-42d3-a9f0-1ccf80e0849a","Type":"ContainerStarted","Data":"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"}
Sep 08 15:27:07.130928 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:07.130799    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:07.130928 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:07.130817    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" event={"ID":"e652b03f-d42d-42d3-a9f0-1ccf80e0849a","Type":"ContainerStarted","Data":"1fdf8a089477846be9b2351a6487d3bea5398455864c3776e52231bde7ae76f8"}
Sep 08 15:27:07.149389 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:07.149342    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" podStartSLOduration=1.149329386 podStartE2EDuration="1.149329386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:27:07.146636533 +0000 UTC m=+868.989433755" watchObservedRunningTime="2026-09-08 15:27:07.149329386 +0000 UTC m=+868.992126608"
Sep 08 15:27:13.600242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:13.600203    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:27:16.034068 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.033966    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" podUID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" containerName="main" containerID="cri-o://63af077a7e3538d8bdcaedf173c275ccdda741185b164c69f3a46f697a07dc92" gracePeriod=45
Sep 08 15:27:16.190219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.190182    2568 generic.go:356] "Generic (PLEG): container finished" podID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" containerID="63af077a7e3538d8bdcaedf173c275ccdda741185b164c69f3a46f697a07dc92" exitCode=0
Sep 08 15:27:16.190386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.190240    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" event={"ID":"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb","Type":"ContainerDied","Data":"63af077a7e3538d8bdcaedf173c275ccdda741185b164c69f3a46f697a07dc92"}
Sep 08 15:27:16.285207 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.285152    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:27:16.424417 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424374    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424417 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424418    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424452    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7255j\" (UniqueName: \"kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424481    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424530    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424659 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424555    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir\") pod \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\" (UID: \"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb\") "
Sep 08 15:27:16.424858 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424687    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home" (OuterVolumeSpecName: "home") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:16.424917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424850    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:16.424917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424890    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:16.425014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424936    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:16.425014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.424952    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:16.426701 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.426671    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm" (OuterVolumeSpecName: "dshm") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:16.426810 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.426725    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:27:16.426810 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.426773    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j" (OuterVolumeSpecName: "kube-api-access-7255j") pod "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" (UID: "f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb"). InnerVolumeSpecName "kube-api-access-7255j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:27:16.526050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.526012    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7255j\" (UniqueName: \"kubernetes.io/projected/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-kube-api-access-7255j\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:16.526050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.526047    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:16.526050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.526056    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:16.526050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:16.526065    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:17.197390 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:17.197363    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"
Sep 08 15:27:17.197390 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:17.197378    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8" event={"ID":"f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb","Type":"ContainerDied","Data":"0cfed83cb48f815e96bcbb25a33e7dfd687932951f3908d05e256ad3603e3a0f"}
Sep 08 15:27:17.197850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:17.197426    2568 scope.go:117] "RemoveContainer" containerID="63af077a7e3538d8bdcaedf173c275ccdda741185b164c69f3a46f697a07dc92"
Sep 08 15:27:17.215660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:17.215636    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:27:17.219218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:17.219197    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-98d03a/autores-model-98d03a-kserve-5b45d5486c-cx8s8"]
Sep 08 15:27:18.138844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:18.138811    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:18.676841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:18.676797    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" path="/var/lib/kubelet/pods/f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb/volumes"
Sep 08 15:27:23.335212 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:23.335179    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:27:32.694090 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:32.694012    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:27:35.989059 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:35.989026    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:35.989506 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:35.989276    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" podUID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" containerName="payload-pre-processing" containerID="cri-o://cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f" gracePeriod=30
Sep 08 15:27:36.233746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.233722    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:36.286605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.286496    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume\") pod \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") "
Sep 08 15:27:36.286782 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.286627    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwd8c\" (UniqueName: \"kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c\") pod \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\" (UID: \"e652b03f-d42d-42d3-a9f0-1ccf80e0849a\") "
Sep 08 15:27:36.286868 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.286844    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e652b03f-d42d-42d3-a9f0-1ccf80e0849a" (UID: "e652b03f-d42d-42d3-a9f0-1ccf80e0849a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:27:36.288696 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.288671    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c" (OuterVolumeSpecName: "kube-api-access-gwd8c") pod "e652b03f-d42d-42d3-a9f0-1ccf80e0849a" (UID: "e652b03f-d42d-42d3-a9f0-1ccf80e0849a"). InnerVolumeSpecName "kube-api-access-gwd8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:27:36.302964 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.302935    2568 generic.go:356] "Generic (PLEG): container finished" podID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" containerID="cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f" exitCode=0
Sep 08 15:27:36.303083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.302981    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"
Sep 08 15:27:36.303083 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.303028    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" event={"ID":"e652b03f-d42d-42d3-a9f0-1ccf80e0849a","Type":"ContainerDied","Data":"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"}
Sep 08 15:27:36.303197 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.303095    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d" event={"ID":"e652b03f-d42d-42d3-a9f0-1ccf80e0849a","Type":"ContainerDied","Data":"1fdf8a089477846be9b2351a6487d3bea5398455864c3776e52231bde7ae76f8"}
Sep 08 15:27:36.303197 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.303127    2568 scope.go:117] "RemoveContainer" containerID="cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"
Sep 08 15:27:36.312533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.312518    2568 scope.go:117] "RemoveContainer" containerID="cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"
Sep 08 15:27:36.312786 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:27:36.312769    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f\": container with ID starting with cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f not found: ID does not exist" containerID="cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"
Sep 08 15:27:36.312830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.312794    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f"} err="failed to get container status \"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f\": rpc error: code = NotFound desc = could not find container \"cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f\": container with ID starting with cbab523cf8b96cd0969080e9833a31dfe66656be691ba706961ab2e5cb61833f not found: ID does not exist"
Sep 08 15:27:36.331195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.330437    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:36.331195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.330489    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b5aeacc1-6c76745c94-jvt4d"]
Sep 08 15:27:36.388122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.388094    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:36.388122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.388118    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwd8c\" (UniqueName: \"kubernetes.io/projected/e652b03f-d42d-42d3-a9f0-1ccf80e0849a-kube-api-access-gwd8c\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:36.675711 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:36.675679    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" path="/var/lib/kubelet/pods/e652b03f-d42d-42d3-a9f0-1ccf80e0849a/volumes"
Sep 08 15:27:38.355376 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.355319    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" podUID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" containerName="main" containerID="cri-o://d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d" gracePeriod=45
Sep 08 15:27:38.616263 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.616201    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:38.646060 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.646023    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:27:38.647808 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.647740    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:27:38.708272 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708244    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708280    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxlb8\" (UniqueName: \"kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708337    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708367    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708382    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708425    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home\") pod \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\" (UID: \"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b\") "
Sep 08 15:27:38.708736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708546    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:38.708736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708695    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:38.708736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708727    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home" (OuterVolumeSpecName: "home") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:38.708866 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.708738    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:38.710613 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.710589    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:27:38.710613 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.710604    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8" (OuterVolumeSpecName: "kube-api-access-dxlb8") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "kube-api-access-dxlb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:27:38.710738 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.710610    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm" (OuterVolumeSpecName: "dshm") pod "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" (UID: "57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:38.809167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.809135    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxlb8\" (UniqueName: \"kubernetes.io/projected/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-kube-api-access-dxlb8\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:38.809167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.809164    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:38.809421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.809178    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:38.809421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.809190    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:38.809421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:38.809201    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:39.321072 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.320981    2568 generic.go:356] "Generic (PLEG): container finished" podID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" containerID="d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d" exitCode=0
Sep 08 15:27:39.321072 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.321024    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"
Sep 08 15:27:39.321072 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.321040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" event={"ID":"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b","Type":"ContainerDied","Data":"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"}
Sep 08 15:27:39.321465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.321079    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq" event={"ID":"57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b","Type":"ContainerDied","Data":"ecb437a19f54f22e264ec5de535f89e204f0ffb0d9ec477c7529785a5c68ac3f"}
Sep 08 15:27:39.321465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.321103    2568 scope.go:117] "RemoveContainer" containerID="d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"
Sep 08 15:27:39.330607 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.330588    2568 scope.go:117] "RemoveContainer" containerID="d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"
Sep 08 15:27:39.330907 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:27:39.330888    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d\": container with ID starting with d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d not found: ID does not exist" containerID="d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"
Sep 08 15:27:39.330956 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.330916    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d"} err="failed to get container status \"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d\": rpc error: code = NotFound desc = could not find container \"d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d\": container with ID starting with d0d1c78caf0c0f1957eb5def434e303c18a218352e42a391d1430b995b56158d not found: ID does not exist"
Sep 08 15:27:39.348817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.345493    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:27:39.352940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:39.352916    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-d4257732-kserve-66c4d59b9-vcswq"]
Sep 08 15:27:40.676166 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:40.676120    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" path="/var/lib/kubelet/pods/57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b/volumes"
Sep 08 15:27:43.533152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533112    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533498    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" containerName="main"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533510    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" containerName="main"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533531    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" containerName="payload-pre-processing"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533538    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" containerName="payload-pre-processing"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533554    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" containerName="main"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533563    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" containerName="main"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533642    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2414c2e-8f1f-457e-8dd2-ba016b9fe7eb" containerName="main"
Sep 08 15:27:43.533644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533651    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c1ae89-ef9a-4f4b-a32b-a3f89fe06b4b" containerName="main"
Sep 08 15:27:43.534073 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.533659    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e652b03f-d42d-42d3-a9f0-1ccf80e0849a" containerName="payload-pre-processing"
Sep 08 15:27:43.538685 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.538665    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.545938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.545099    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0c2253-serving-cert\""
Sep 08 15:27:43.547788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.547764    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:27:43.604190 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.604162    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:27:43.652027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.652000    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.652027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.652030    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.652226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.652110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.652226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.652166    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqjm\" (UniqueName: \"kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.752557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.752523    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.752745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.752570    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqjm\" (UniqueName: \"kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.752745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.752650    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.752745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.752679    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.753383 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.753355    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.755106 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.755072    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.755106 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.755083    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.760537 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.760516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqjm\" (UniqueName: \"kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm\") pod \"maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:43.854039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:43.853964    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:44.134042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:44.134015    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:27:44.136340 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:27:44.136315    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7eff190_d483_4b88_bfb5_c25594c92b9b.slice/crio-c1554e78b039541aa19e2584be75eb646447584197a6b27f929a7b8373980557 WatchSource:0}: Error finding container c1554e78b039541aa19e2584be75eb646447584197a6b27f929a7b8373980557: Status 404 returned error can't find the container with id c1554e78b039541aa19e2584be75eb646447584197a6b27f929a7b8373980557
Sep 08 15:27:44.349664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:44.349628    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" event={"ID":"d7eff190-d483-4b88-bfb5-c25594c92b9b","Type":"ContainerStarted","Data":"c1554e78b039541aa19e2584be75eb646447584197a6b27f929a7b8373980557"}
Sep 08 15:27:45.356632 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:45.356598    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" event={"ID":"d7eff190-d483-4b88-bfb5-c25594c92b9b","Type":"ContainerStarted","Data":"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"}
Sep 08 15:27:45.357034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:45.356671    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:45.383649 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:45.383594    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" podStartSLOduration=1.685354107 podStartE2EDuration="2.383580472s" totalImagesPullingTime="698.226365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:27:45.382090579 +0000 UTC m=+907.224887803" watchObservedRunningTime="2026-09-08 15:27:45.383580472 +0000 UTC m=+907.226377695"
Sep 08 15:27:48.001547 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:48.001511    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:27:51.367570 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:51.367530    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:27:58.624129 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.624084    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" podUID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" containerName="main" containerID="cri-o://8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616" gracePeriod=45
Sep 08 15:27:58.886706 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.886640    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:27:58.977418 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977390    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6ch9\" (UniqueName: \"kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977437    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977463    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977487    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977522    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977546    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs\") pod \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\" (UID: \"d6cb2376-48b8-4fa8-9559-239e2c40bb0f\") "
Sep 08 15:27:58.977841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977709    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:58.977841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977769    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:58.977841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977825    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home" (OuterVolumeSpecName: "home") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:58.977996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977910    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:58.977996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977924    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:58.977996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.977939    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:58.979677 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.979647    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9" (OuterVolumeSpecName: "kube-api-access-t6ch9") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "kube-api-access-t6ch9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:27:58.979809 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.979696    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm" (OuterVolumeSpecName: "dshm") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:27:58.979809 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:58.979693    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6cb2376-48b8-4fa8-9559-239e2c40bb0f" (UID: "d6cb2376-48b8-4fa8-9559-239e2c40bb0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:27:59.079187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.079161    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6ch9\" (UniqueName: \"kubernetes.io/projected/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-kube-api-access-t6ch9\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:59.079187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.079182    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:59.079187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.079191    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6cb2376-48b8-4fa8-9559-239e2c40bb0f-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:27:59.437462 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.437427    2568 generic.go:356] "Generic (PLEG): container finished" podID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" containerID="8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616" exitCode=0
Sep 08 15:27:59.437625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.437474    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"
Sep 08 15:27:59.437625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.437481    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" event={"ID":"d6cb2376-48b8-4fa8-9559-239e2c40bb0f","Type":"ContainerDied","Data":"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"}
Sep 08 15:27:59.437625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.437583    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts" event={"ID":"d6cb2376-48b8-4fa8-9559-239e2c40bb0f","Type":"ContainerDied","Data":"0c51fcfb36f9244267eba243a90ac5041c1efefd66c357c7e090046359a4a097"}
Sep 08 15:27:59.437625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.437599    2568 scope.go:117] "RemoveContainer" containerID="8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"
Sep 08 15:27:59.447145 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.447129    2568 scope.go:117] "RemoveContainer" containerID="8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"
Sep 08 15:27:59.447405 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:27:59.447388    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616\": container with ID starting with 8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616 not found: ID does not exist" containerID="8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"
Sep 08 15:27:59.447471 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.447417    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616"} err="failed to get container status \"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616\": rpc error: code = NotFound desc = could not find container \"8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616\": container with ID starting with 8866efeb09bbdbf666ccb32ea9c1b184973767f1be48949aaf1a46140a8c7616 not found: ID does not exist"
Sep 08 15:27:59.459421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.459399    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:27:59.462959 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:27:59.462938    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-d4257732-kserve-5bc85d8b95-sfvts"]
Sep 08 15:28:00.675856 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:00.675818    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" path="/var/lib/kubelet/pods/d6cb2376-48b8-4fa8-9559-239e2c40bb0f/volumes"
Sep 08 15:28:01.494806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:01.494774    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:28:15.118958 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.118925    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:28:15.119427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.119348    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" containerName="main"
Sep 08 15:28:15.119427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.119366    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" containerName="main"
Sep 08 15:28:15.119503 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.119462    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6cb2376-48b8-4fa8-9559-239e2c40bb0f" containerName="main"
Sep 08 15:28:15.122607 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.122591    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.125668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.125641    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b75273\"/\"default-dockercfg-z4pzd\""
Sep 08 15:28:15.125794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.125669    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b75273\"/\"kube-root-ca.crt\""
Sep 08 15:28:15.125794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.125642    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b75273\"/\"openshift-service-ca.crt\""
Sep 08 15:28:15.125794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.125694    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b75273\"/\"test-model-b75273-kserve-self-signed-certs\""
Sep 08 15:28:15.132388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.132367    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:28:15.214910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.214881    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.214910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.214910    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.215068 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.214942    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.215068 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.215040    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4fj\" (UniqueName: \"kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.215151 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.215070    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.215151 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.215100    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316339 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316286    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4fj\" (UniqueName: \"kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316346    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316365    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316413    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316429    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316475    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316811 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316790    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316852    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.316936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.316922    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.318690 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.318667    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.318957 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.318941    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.325205 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.325181    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4fj\" (UniqueName: \"kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj\") pod \"test-model-b75273-kserve-7584fdbcf7-wkvj4\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") " pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.434246 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.434175    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:15.705861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.705836    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:28:15.707764 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:28:15.707737    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e5ec621_d3a9_4239_a30c_a288362e4585.slice/crio-b9311093ddb4b000eb9d06daf2ca195c4ed04e5bd80089a54536b4d14aead4e1 WatchSource:0}: Error finding container b9311093ddb4b000eb9d06daf2ca195c4ed04e5bd80089a54536b4d14aead4e1: Status 404 returned error can't find the container with id b9311093ddb4b000eb9d06daf2ca195c4ed04e5bd80089a54536b4d14aead4e1
Sep 08 15:28:15.808164 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:15.808134    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:28:16.536611 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:16.536575    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" event={"ID":"2e5ec621-d3a9-4239-a30c-a288362e4585","Type":"ContainerStarted","Data":"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"}
Sep 08 15:28:16.536979 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:16.536622    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" event={"ID":"2e5ec621-d3a9-4239-a30c-a288362e4585","Type":"ContainerStarted","Data":"b9311093ddb4b000eb9d06daf2ca195c4ed04e5bd80089a54536b4d14aead4e1"}
Sep 08 15:28:16.554551 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:16.554503    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" podStartSLOduration=1.55448806 podStartE2EDuration="1.55448806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:16.551854428 +0000 UTC m=+938.394651664" watchObservedRunningTime="2026-09-08 15:28:16.55448806 +0000 UTC m=+938.397285292"
Sep 08 15:28:21.910860 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.910822    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:21.914518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.914499    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:21.917667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.917642    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0c2253\"/\"default-dockercfg-h9wd5\""
Sep 08 15:28:21.917667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.917661    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0c2253\"/\"openshift-service-ca.crt\""
Sep 08 15:28:21.917854 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.917645    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0c2253\"/\"explicit-model-0c2253-kserve-self-signed-certs\""
Sep 08 15:28:21.917854 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.917677    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0c2253\"/\"kube-root-ca.crt\""
Sep 08 15:28:21.924973 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:21.924953    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:22.075247 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075219    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.075430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075259    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.075430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075287    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cf6n\" (UniqueName: \"kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.075430 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075373    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.075543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.075543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.075483    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176129    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176170    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176312    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176370    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176411    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cf6n\" (UniqueName: \"kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176466    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176608    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176718 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.176777 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.176715    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.178535 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.178516    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.178928 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.178908    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.183896 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.183875    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cf6n\" (UniqueName: \"kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n\") pod \"explicit-model-0c2253-kserve-55fb8d55bc-zjfsn\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") " pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.224921 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.224901    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:22.495858 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.495836    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:22.497182 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:28:22.497159    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0295a51_2045_4295_9294_e116df634259.slice/crio-43ad7b6817f53ea2b203f705088d5de6fc11641ee35f5ee44c002a3a09813d8c WatchSource:0}: Error finding container 43ad7b6817f53ea2b203f705088d5de6fc11641ee35f5ee44c002a3a09813d8c: Status 404 returned error can't find the container with id 43ad7b6817f53ea2b203f705088d5de6fc11641ee35f5ee44c002a3a09813d8c
Sep 08 15:28:22.573961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.573931    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" event={"ID":"d0295a51-2045-4295-9294-e116df634259","Type":"ContainerStarted","Data":"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"}
Sep 08 15:28:22.574089 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.573981    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" event={"ID":"d0295a51-2045-4295-9294-e116df634259","Type":"ContainerStarted","Data":"43ad7b6817f53ea2b203f705088d5de6fc11641ee35f5ee44c002a3a09813d8c"}
Sep 08 15:28:22.591552 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:22.591506    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" podStartSLOduration=1.591492039 podStartE2EDuration="1.591492039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:28:22.588918562 +0000 UTC m=+944.431715785" watchObservedRunningTime="2026-09-08 15:28:22.591492039 +0000 UTC m=+944.434289262"
Sep 08 15:28:25.435244 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:25.435200    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:25.435746 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:25.435265    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:25.447748 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:25.447722    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:25.601125 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:25.601095    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:28:28.710827 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:28.710793    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:28:29.214640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:29.214606    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:32.225566 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:32.225532    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:39.939035 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:39.938998    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:28:39.939448 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:39.939196    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" podUID="d7eff190-d483-4b88-bfb5-c25594c92b9b" containerName="maas-api" containerID="cri-o://fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357" gracePeriod=30
Sep 08 15:28:40.197331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.197241    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:28:40.226670 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.226642    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca\") pod \"d7eff190-d483-4b88-bfb5-c25594c92b9b\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") "
Sep 08 15:28:40.226818 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.226678    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs\") pod \"d7eff190-d483-4b88-bfb5-c25594c92b9b\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") "
Sep 08 15:28:40.226818 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.226800    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls\") pod \"d7eff190-d483-4b88-bfb5-c25594c92b9b\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") "
Sep 08 15:28:40.226940 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.226859    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpqjm\" (UniqueName: \"kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm\") pod \"d7eff190-d483-4b88-bfb5-c25594c92b9b\" (UID: \"d7eff190-d483-4b88-bfb5-c25594c92b9b\") "
Sep 08 15:28:40.227097 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.227035    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d7eff190-d483-4b88-bfb5-c25594c92b9b" (UID: "d7eff190-d483-4b88-bfb5-c25594c92b9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:28:40.227239 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.227149    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7eff190-d483-4b88-bfb5-c25594c92b9b-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:40.229030 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.229000    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d7eff190-d483-4b88-bfb5-c25594c92b9b" (UID: "d7eff190-d483-4b88-bfb5-c25594c92b9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:28:40.229138 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.229096    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm" (OuterVolumeSpecName: "kube-api-access-jpqjm") pod "d7eff190-d483-4b88-bfb5-c25594c92b9b" (UID: "d7eff190-d483-4b88-bfb5-c25594c92b9b"). InnerVolumeSpecName "kube-api-access-jpqjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:28:40.229186 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.229146    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d7eff190-d483-4b88-bfb5-c25594c92b9b" (UID: "d7eff190-d483-4b88-bfb5-c25594c92b9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:28:40.328289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.328256    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:40.328289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.328282    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7eff190-d483-4b88-bfb5-c25594c92b9b-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:40.328289 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.328313    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqjm\" (UniqueName: \"kubernetes.io/projected/d7eff190-d483-4b88-bfb5-c25594c92b9b-kube-api-access-jpqjm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:40.676188 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.676150    2568 generic.go:356] "Generic (PLEG): container finished" podID="d7eff190-d483-4b88-bfb5-c25594c92b9b" containerID="fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357" exitCode=0
Sep 08 15:28:40.676402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.676197    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" event={"ID":"d7eff190-d483-4b88-bfb5-c25594c92b9b","Type":"ContainerDied","Data":"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"}
Sep 08 15:28:40.676402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.676207    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"
Sep 08 15:28:40.676402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.676231    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t" event={"ID":"d7eff190-d483-4b88-bfb5-c25594c92b9b","Type":"ContainerDied","Data":"c1554e78b039541aa19e2584be75eb646447584197a6b27f929a7b8373980557"}
Sep 08 15:28:40.676402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.676258    2568 scope.go:117] "RemoveContainer" containerID="fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"
Sep 08 15:28:40.686394 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.686331    2568 scope.go:117] "RemoveContainer" containerID="fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"
Sep 08 15:28:40.686673 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:28:40.686639    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357\": container with ID starting with fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357 not found: ID does not exist" containerID="fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"
Sep 08 15:28:40.686749 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.686680    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357"} err="failed to get container status \"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357\": rpc error: code = NotFound desc = could not find container \"fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357\": container with ID starting with fb7ea83efd57c36fddcea4862a99ae004b94ab783155c3bbf789caff1fb31357 not found: ID does not exist"
Sep 08 15:28:40.701557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.701534    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:28:40.705484 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:40.705463    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0c2253-6f9d5f6bdd-5kz2t"]
Sep 08 15:28:42.676486 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:42.676455    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7eff190-d483-4b88-bfb5-c25594c92b9b" path="/var/lib/kubelet/pods/d7eff190-d483-4b88-bfb5-c25594c92b9b/volumes"
Sep 08 15:28:43.697625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:43.697592    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:28:44.236185 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.236122    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" podUID="d0295a51-2045-4295-9294-e116df634259" containerName="main" containerID="cri-o://0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6" gracePeriod=45
Sep 08 15:28:44.492276 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.492206    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:44.563850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.563818    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.563850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.563853    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cf6n\" (UniqueName: \"kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.564111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.563872    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.564111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.563957    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.564111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.563993    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.564111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564103    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir\") pod \"d0295a51-2045-4295-9294-e116df634259\" (UID: \"d0295a51-2045-4295-9294-e116df634259\") "
Sep 08 15:28:44.564437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564228    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home" (OuterVolumeSpecName: "home") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:28:44.564437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564325    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:28:44.564437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564395    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:28:44.564437 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564405    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.564649 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.564438    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.566165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.566138    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm" (OuterVolumeSpecName: "dshm") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:28:44.566278 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.566209    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n" (OuterVolumeSpecName: "kube-api-access-2cf6n") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "kube-api-access-2cf6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:28:44.566278 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.566214    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0295a51-2045-4295-9294-e116df634259" (UID: "d0295a51-2045-4295-9294-e116df634259"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:28:44.665679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.665649    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.665679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.665676    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0295a51-2045-4295-9294-e116df634259-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.665853 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.665685    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cf6n\" (UniqueName: \"kubernetes.io/projected/d0295a51-2045-4295-9294-e116df634259-kube-api-access-2cf6n\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.665853 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.665694    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0295a51-2045-4295-9294-e116df634259-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:28:44.708472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.708437    2568 generic.go:356] "Generic (PLEG): container finished" podID="d0295a51-2045-4295-9294-e116df634259" containerID="0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6" exitCode=0
Sep 08 15:28:44.708806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.708495    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"
Sep 08 15:28:44.708806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.708519    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" event={"ID":"d0295a51-2045-4295-9294-e116df634259","Type":"ContainerDied","Data":"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"}
Sep 08 15:28:44.708806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.708578    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn" event={"ID":"d0295a51-2045-4295-9294-e116df634259","Type":"ContainerDied","Data":"43ad7b6817f53ea2b203f705088d5de6fc11641ee35f5ee44c002a3a09813d8c"}
Sep 08 15:28:44.708806 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.708599    2568 scope.go:117] "RemoveContainer" containerID="0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"
Sep 08 15:28:44.717288 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.717269    2568 scope.go:117] "RemoveContainer" containerID="0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"
Sep 08 15:28:44.717595 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:28:44.717576    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6\": container with ID starting with 0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6 not found: ID does not exist" containerID="0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"
Sep 08 15:28:44.717651 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.717606    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6"} err="failed to get container status \"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6\": rpc error: code = NotFound desc = could not find container \"0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6\": container with ID starting with 0f2299425cfe2631706b25f36edb81897fef62d52cf189ec3a387009f3c5ece6 not found: ID does not exist"
Sep 08 15:28:44.723796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.723775    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:44.729633 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:44.729613    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0c2253/explicit-model-0c2253-kserve-55fb8d55bc-zjfsn"]
Sep 08 15:28:46.675359 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:46.675316    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0295a51-2045-4295-9294-e116df634259" path="/var/lib/kubelet/pods/d0295a51-2045-4295-9294-e116df634259/volumes"
Sep 08 15:28:58.302657 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:28:58.302619    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:29:00.514712 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.514680    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:00.515161 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515065    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7eff190-d483-4b88-bfb5-c25594c92b9b" containerName="maas-api"
Sep 08 15:29:00.515161 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515078    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7eff190-d483-4b88-bfb5-c25594c92b9b" containerName="maas-api"
Sep 08 15:29:00.515161 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515088    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0295a51-2045-4295-9294-e116df634259" containerName="main"
Sep 08 15:29:00.515161 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515093    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0295a51-2045-4295-9294-e116df634259" containerName="main"
Sep 08 15:29:00.515324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515167    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7eff190-d483-4b88-bfb5-c25594c92b9b" containerName="maas-api"
Sep 08 15:29:00.515324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.515178    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0295a51-2045-4295-9294-e116df634259" containerName="main"
Sep 08 15:29:00.519648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.519632    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.523190 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.523167    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e8425d\"/\"kube-root-ca.crt\""
Sep 08 15:29:00.523313 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.523213    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e8425d\"/\"noten-model-e8425d-kserve-self-signed-certs\""
Sep 08 15:29:00.523313 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.523167    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e8425d\"/\"openshift-service-ca.crt\""
Sep 08 15:29:00.523313 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.523269    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e8425d\"/\"default-dockercfg-76xpd\""
Sep 08 15:29:00.533870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.533850    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:00.597087 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597055    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5gj\" (UniqueName: \"kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.597219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597116    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.597219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597190    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.597333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597267    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.597333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597320    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.597405 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.597356    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698781 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698749    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698781 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698780    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698809    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698858    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5gj\" (UniqueName: \"kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698892    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.698987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.698940    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.699215 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.699179    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.699339 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.699320    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.699339 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.699333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.701081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.701060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.701245 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.701227    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.706219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.706197    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5gj\" (UniqueName: \"kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj\") pod \"noten-model-e8425d-kserve-58b49b659-z9c9x\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") " pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:00.829040 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:00.828957    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:01.104218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:01.104109    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:01.107286 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:29:01.107251    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6541934e_ae65_47bb_9cfe_8116fcce6421.slice/crio-0bc2b1773b8761bcb707193cbae98f79aa5d43a8622e9639959fe2d9a2d1a4f8 WatchSource:0}: Error finding container 0bc2b1773b8761bcb707193cbae98f79aa5d43a8622e9639959fe2d9a2d1a4f8: Status 404 returned error can't find the container with id 0bc2b1773b8761bcb707193cbae98f79aa5d43a8622e9639959fe2d9a2d1a4f8
Sep 08 15:29:01.801526 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:01.801492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" event={"ID":"6541934e-ae65-47bb-9cfe-8116fcce6421","Type":"ContainerStarted","Data":"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"}
Sep 08 15:29:01.801919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:01.801535    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" event={"ID":"6541934e-ae65-47bb-9cfe-8116fcce6421","Type":"ContainerStarted","Data":"0bc2b1773b8761bcb707193cbae98f79aa5d43a8622e9639959fe2d9a2d1a4f8"}
Sep 08 15:29:01.821606 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:01.821561    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" podStartSLOduration=1.821545864 podStartE2EDuration="1.821545864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:29:01.819768301 +0000 UTC m=+983.662565539" watchObservedRunningTime="2026-09-08 15:29:01.821545864 +0000 UTC m=+983.664343086"
Sep 08 15:29:08.504582 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:08.504548    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:10.829716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:10.829682    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:11.500352 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:11.500322    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:29:23.524997 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.524921    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" podUID="6541934e-ae65-47bb-9cfe-8116fcce6421" containerName="main" containerID="cri-o://d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515" gracePeriod=45
Sep 08 15:29:23.772919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.772890    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:23.897427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897394    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897446    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897474    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm5gj\" (UniqueName: \"kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897496    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897516    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897619 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897541    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache\") pod \"6541934e-ae65-47bb-9cfe-8116fcce6421\" (UID: \"6541934e-ae65-47bb-9cfe-8116fcce6421\") "
Sep 08 15:29:23.897894 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.897697    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:23.898208 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.898183    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home" (OuterVolumeSpecName: "home") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:23.898334 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.898286    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache" (OuterVolumeSpecName: "model-cache") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:23.898402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.898330    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:23.901939 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.900467    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm" (OuterVolumeSpecName: "dshm") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:23.901939 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.900676    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj" (OuterVolumeSpecName: "kube-api-access-cm5gj") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "kube-api-access-cm5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:29:23.907680 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.907646    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6541934e-ae65-47bb-9cfe-8116fcce6421" (UID: "6541934e-ae65-47bb-9cfe-8116fcce6421"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:29:23.932828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.932788    2568 generic.go:356] "Generic (PLEG): container finished" podID="6541934e-ae65-47bb-9cfe-8116fcce6421" containerID="d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515" exitCode=0
Sep 08 15:29:23.932969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.932850    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" event={"ID":"6541934e-ae65-47bb-9cfe-8116fcce6421","Type":"ContainerDied","Data":"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"}
Sep 08 15:29:23.932969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.932912    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"
Sep 08 15:29:23.932969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.932954    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x" event={"ID":"6541934e-ae65-47bb-9cfe-8116fcce6421","Type":"ContainerDied","Data":"0bc2b1773b8761bcb707193cbae98f79aa5d43a8622e9639959fe2d9a2d1a4f8"}
Sep 08 15:29:23.933136 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.932988    2568 scope.go:117] "RemoveContainer" containerID="d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"
Sep 08 15:29:23.943910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.943892    2568 scope.go:117] "RemoveContainer" containerID="d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"
Sep 08 15:29:23.944257 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:29:23.944229    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515\": container with ID starting with d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515 not found: ID does not exist" containerID="d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"
Sep 08 15:29:23.944402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.944264    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515"} err="failed to get container status \"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515\": rpc error: code = NotFound desc = could not find container \"d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515\": container with ID starting with d37f6ac869cc3d1e629482899f73c23e2ddbc832baaae6a927607e7446c5f515 not found: ID does not exist"
Sep 08 15:29:23.954770 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.954745    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:23.960402 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.960377    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e8425d/noten-model-e8425d-kserve-58b49b659-z9c9x"]
Sep 08 15:29:23.998958 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.998933    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:23.998958 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.998957    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5gj\" (UniqueName: \"kubernetes.io/projected/6541934e-ae65-47bb-9cfe-8116fcce6421-kube-api-access-cm5gj\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:23.999114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.998967    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:23.999114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.998978    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6541934e-ae65-47bb-9cfe-8116fcce6421-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:23.999114 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:23.998987    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6541934e-ae65-47bb-9cfe-8116fcce6421-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:24.676134 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:24.676099    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6541934e-ae65-47bb-9cfe-8116fcce6421" path="/var/lib/kubelet/pods/6541934e-ae65-47bb-9cfe-8116fcce6421/volumes"
Sep 08 15:29:25.652037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:25.652005    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:29:29.400831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:29.400797    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:29:39.598596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:39.598559    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:29:40.672198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:40.672161    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" podUID="2e5ec621-d3a9-4239-a30c-a288362e4585" containerName="main" containerID="cri-o://2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80" gracePeriod=45
Sep 08 15:29:40.926442 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:40.926383    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:29:41.028763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.028727    2568 generic.go:356] "Generic (PLEG): container finished" podID="2e5ec621-d3a9-4239-a30c-a288362e4585" containerID="2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80" exitCode=0
Sep 08 15:29:41.028763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.028766    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"
Sep 08 15:29:41.029013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.028771    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" event={"ID":"2e5ec621-d3a9-4239-a30c-a288362e4585","Type":"ContainerDied","Data":"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"}
Sep 08 15:29:41.029013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.028807    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4" event={"ID":"2e5ec621-d3a9-4239-a30c-a288362e4585","Type":"ContainerDied","Data":"b9311093ddb4b000eb9d06daf2ca195c4ed04e5bd80089a54536b4d14aead4e1"}
Sep 08 15:29:41.029013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.028829    2568 scope.go:117] "RemoveContainer" containerID="2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"
Sep 08 15:29:41.036403 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036376    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036522 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036428    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036522 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036473    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036522 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036516    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036553    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx4fj\" (UniqueName: \"kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs\") pod \"2e5ec621-d3a9-4239-a30c-a288362e4585\" (UID: \"2e5ec621-d3a9-4239-a30c-a288362e4585\") "
Sep 08 15:29:41.036694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036661    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:41.036847 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036737    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home" (OuterVolumeSpecName: "home") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:41.036847 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036754    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:41.036960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036914    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.036960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036932    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.036960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.036941    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.037739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.037719    2568 scope.go:117] "RemoveContainer" containerID="2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"
Sep 08 15:29:41.038087 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:29:41.038065    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80\": container with ID starting with 2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80 not found: ID does not exist" containerID="2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"
Sep 08 15:29:41.038163 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.038103    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80"} err="failed to get container status \"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80\": rpc error: code = NotFound desc = could not find container \"2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80\": container with ID starting with 2a181a89051316765899a84beefc23abae0501eb3050f653acf71f8c3bb03a80 not found: ID does not exist"
Sep 08 15:29:41.038687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.038664    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm" (OuterVolumeSpecName: "dshm") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:29:41.038777 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.038743    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj" (OuterVolumeSpecName: "kube-api-access-zx4fj") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "kube-api-access-zx4fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:29:41.038837 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.038794    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e5ec621-d3a9-4239-a30c-a288362e4585" (UID: "2e5ec621-d3a9-4239-a30c-a288362e4585"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:29:41.138258 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.138227    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e5ec621-d3a9-4239-a30c-a288362e4585-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.138258 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.138252    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e5ec621-d3a9-4239-a30c-a288362e4585-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.138258 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.138264    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx4fj\" (UniqueName: \"kubernetes.io/projected/2e5ec621-d3a9-4239-a30c-a288362e4585-kube-api-access-zx4fj\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:41.356674 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.356645    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:29:41.361416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:41.361389    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b75273/test-model-b75273-kserve-7584fdbcf7-wkvj4"]
Sep 08 15:29:42.680667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:42.680622    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e5ec621-d3a9-4239-a30c-a288362e4585" path="/var/lib/kubelet/pods/2e5ec621-d3a9-4239-a30c-a288362e4585/volumes"
Sep 08 15:29:53.903214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:53.903177    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:29:53.903736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:53.903442    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" podUID="7828e655-c838-442e-a454-fe670d8c45dc" containerName="maas-api" containerID="cri-o://75ac0fcf22509adc975fb676c36a8e8516029202381e1291add84825071fc7fb" gracePeriod=30
Sep 08 15:29:53.957394 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:53.957361    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:29:53.957636 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:53.957590    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" podUID="5214ae18-243f-461f-bc2f-e118a8e46242" containerName="payload-pre-processing" containerID="cri-o://d167c2cee1793e3fae6fbf2538d5dad5cf4d2a42f9cbd2204ff7d964bfec26aa" gracePeriod=30
Sep 08 15:29:54.104514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.104468    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" event={"ID":"7828e655-c838-442e-a454-fe670d8c45dc","Type":"ContainerDied","Data":"75ac0fcf22509adc975fb676c36a8e8516029202381e1291add84825071fc7fb"}
Sep 08 15:29:54.104941 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.104392    2568 generic.go:356] "Generic (PLEG): container finished" podID="7828e655-c838-442e-a454-fe670d8c45dc" containerID="75ac0fcf22509adc975fb676c36a8e8516029202381e1291add84825071fc7fb" exitCode=0
Sep 08 15:29:54.107057 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.107029    2568 generic.go:356] "Generic (PLEG): container finished" podID="5214ae18-243f-461f-bc2f-e118a8e46242" containerID="d167c2cee1793e3fae6fbf2538d5dad5cf4d2a42f9cbd2204ff7d964bfec26aa" exitCode=0
Sep 08 15:29:54.107186 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.107087    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" event={"ID":"5214ae18-243f-461f-bc2f-e118a8e46242","Type":"ContainerDied","Data":"d167c2cee1793e3fae6fbf2538d5dad5cf4d2a42f9cbd2204ff7d964bfec26aa"}
Sep 08 15:29:54.216487 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.216464    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:29:54.241071 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.241009    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:29:54.358594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358563    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume\") pod \"5214ae18-243f-461f-bc2f-e118a8e46242\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") "
Sep 08 15:29:54.358771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358607    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls\") pod \"7828e655-c838-442e-a454-fe670d8c45dc\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") "
Sep 08 15:29:54.358771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358653    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca\") pod \"7828e655-c838-442e-a454-fe670d8c45dc\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") "
Sep 08 15:29:54.358771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358671    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trnjk\" (UniqueName: \"kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk\") pod \"5214ae18-243f-461f-bc2f-e118a8e46242\" (UID: \"5214ae18-243f-461f-bc2f-e118a8e46242\") "
Sep 08 15:29:54.358771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358696    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs\") pod \"7828e655-c838-442e-a454-fe670d8c45dc\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") "
Sep 08 15:29:54.358771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358772    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mlpz\" (UniqueName: \"kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz\") pod \"7828e655-c838-442e-a454-fe670d8c45dc\" (UID: \"7828e655-c838-442e-a454-fe670d8c45dc\") "
Sep 08 15:29:54.359047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.358982    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5214ae18-243f-461f-bc2f-e118a8e46242" (UID: "5214ae18-243f-461f-bc2f-e118a8e46242"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:29:54.359047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.359033    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7828e655-c838-442e-a454-fe670d8c45dc" (UID: "7828e655-c838-442e-a454-fe670d8c45dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:29:54.360906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.360867    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7828e655-c838-442e-a454-fe670d8c45dc" (UID: "7828e655-c838-442e-a454-fe670d8c45dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:29:54.361009 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.360920    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk" (OuterVolumeSpecName: "kube-api-access-trnjk") pod "5214ae18-243f-461f-bc2f-e118a8e46242" (UID: "5214ae18-243f-461f-bc2f-e118a8e46242"). InnerVolumeSpecName "kube-api-access-trnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:29:54.361009 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.360939    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz" (OuterVolumeSpecName: "kube-api-access-8mlpz") pod "7828e655-c838-442e-a454-fe670d8c45dc" (UID: "7828e655-c838-442e-a454-fe670d8c45dc"). InnerVolumeSpecName "kube-api-access-8mlpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:29:54.361009 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.360944    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7828e655-c838-442e-a454-fe670d8c45dc" (UID: "7828e655-c838-442e-a454-fe670d8c45dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460016    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trnjk\" (UniqueName: \"kubernetes.io/projected/5214ae18-243f-461f-bc2f-e118a8e46242-kube-api-access-trnjk\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460038    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460047    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mlpz\" (UniqueName: \"kubernetes.io/projected/7828e655-c838-442e-a454-fe670d8c45dc-kube-api-access-8mlpz\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460057    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5214ae18-243f-461f-bc2f-e118a8e46242-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460066    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7828e655-c838-442e-a454-fe670d8c45dc-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:54.460079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:54.460074    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7828e655-c838-442e-a454-fe670d8c45dc-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:29:55.113246 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.113217    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"
Sep 08 15:29:55.113723 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.113242    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4" event={"ID":"5214ae18-243f-461f-bc2f-e118a8e46242","Type":"ContainerDied","Data":"6c51cd1477a5b1c642bbc2bc16b883dc9d8455899a20f292115d8c34cb258080"}
Sep 08 15:29:55.113723 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.113358    2568 scope.go:117] "RemoveContainer" containerID="d167c2cee1793e3fae6fbf2538d5dad5cf4d2a42f9cbd2204ff7d964bfec26aa"
Sep 08 15:29:55.115108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.115081    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt" event={"ID":"7828e655-c838-442e-a454-fe670d8c45dc","Type":"ContainerDied","Data":"f52b14dc5dfa1ce620b29ff2ddf88ddcdcd1ae2279bcbd7483ad912d2ba5e558"}
Sep 08 15:29:55.115224 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.115110    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"
Sep 08 15:29:55.122615 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.122597    2568 scope.go:117] "RemoveContainer" containerID="75ac0fcf22509adc975fb676c36a8e8516029202381e1291add84825071fc7fb"
Sep 08 15:29:55.132281 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.132203    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:29:55.134699 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.134678    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b75273-79cfc96b6f-8tprt"]
Sep 08 15:29:55.144597 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.144576    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:29:55.150491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:55.150472    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b75273-9c79cfd94-vxzr4"]
Sep 08 15:29:56.678363 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:56.678330    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5214ae18-243f-461f-bc2f-e118a8e46242" path="/var/lib/kubelet/pods/5214ae18-243f-461f-bc2f-e118a8e46242/volumes"
Sep 08 15:29:56.678745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:56.678716    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7828e655-c838-442e-a454-fe670d8c45dc" path="/var/lib/kubelet/pods/7828e655-c838-442e-a454-fe670d8c45dc/volumes"
Sep 08 15:29:56.701841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:29:56.701811    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:30:05.948829 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.948797    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:30:05.949371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949350    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7828e655-c838-442e-a454-fe670d8c45dc" containerName="maas-api"
Sep 08 15:30:05.949371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949373    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7828e655-c838-442e-a454-fe670d8c45dc" containerName="maas-api"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949387    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6541934e-ae65-47bb-9cfe-8116fcce6421" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949396    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6541934e-ae65-47bb-9cfe-8116fcce6421" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949407    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5214ae18-243f-461f-bc2f-e118a8e46242" containerName="payload-pre-processing"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949415    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5214ae18-243f-461f-bc2f-e118a8e46242" containerName="payload-pre-processing"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949432    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e5ec621-d3a9-4239-a30c-a288362e4585" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949440    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e5ec621-d3a9-4239-a30c-a288362e4585" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949535    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6541934e-ae65-47bb-9cfe-8116fcce6421" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949549    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e5ec621-d3a9-4239-a30c-a288362e4585" containerName="main"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949560    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5214ae18-243f-461f-bc2f-e118a8e46242" containerName="payload-pre-processing"
Sep 08 15:30:05.949586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.949572    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7828e655-c838-442e-a454-fe670d8c45dc" containerName="maas-api"
Sep 08 15:30:05.952893 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.952870    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:05.955192 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.955170    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:30:05.955316 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.955180    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-fc867c-serving-cert\""
Sep 08 15:30:05.955316 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.955189    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4m2fn\""
Sep 08 15:30:05.955924 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.955906    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:30:05.956037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.956019    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:30:05.965775 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:05.965753    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:30:06.058594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.058561    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.058763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.058612    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.058763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.058633    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.058842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.058757    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2xg\" (UniqueName: \"kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.159427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.159386    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.159640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.159442    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.159640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.159539    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2xg\" (UniqueName: \"kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.159640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.159567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.160236 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.160214    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.161950 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.161927    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.162039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.162021    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.173133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.173110    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2xg\" (UniqueName: \"kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg\") pod \"maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.265586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.265496    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:06.545066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.545037    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:30:06.546862 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:30:06.546826    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951300b2_7a58_407d_bbca_2e8060822b96.slice/crio-8e39a5d309eb185d0a14d2f1055168f68cca5e658e9f2f45027f8a8f25b46794 WatchSource:0}: Error finding container 8e39a5d309eb185d0a14d2f1055168f68cca5e658e9f2f45027f8a8f25b46794: Status 404 returned error can't find the container with id 8e39a5d309eb185d0a14d2f1055168f68cca5e658e9f2f45027f8a8f25b46794
Sep 08 15:30:06.548281 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.548263    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:30:06.846133 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.846054    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:30:06.850513 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.850498    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:06.853965 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.853942    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fc867c\""
Sep 08 15:30:06.854097 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.853992    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fc867c-dockercfg-2c956\""
Sep 08 15:30:06.860111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.860084    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:30:06.969030 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.968999    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th87v\" (UniqueName: \"kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:06.969477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:06.969037    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.070014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.069990    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th87v\" (UniqueName: \"kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.070130 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.070024    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.070736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.070715    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.077723 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.077696    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th87v\" (UniqueName: \"kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v\") pod \"payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.162764 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.162725    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:07.190627 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.190582    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" event={"ID":"951300b2-7a58-407d-bbca-2e8060822b96","Type":"ContainerStarted","Data":"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"}
Sep 08 15:30:07.190743 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.190653    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" event={"ID":"951300b2-7a58-407d-bbca-2e8060822b96","Type":"ContainerStarted","Data":"8e39a5d309eb185d0a14d2f1055168f68cca5e658e9f2f45027f8a8f25b46794"}
Sep 08 15:30:07.190743 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.190702    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:07.223536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.223485    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" podStartSLOduration=1.750660147 podStartE2EDuration="2.22346914s" totalImagesPullingTime="472.808993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:07.215438933 +0000 UTC m=+1049.058236157" watchObservedRunningTime="2026-09-08 15:30:07.22346914 +0000 UTC m=+1049.066266363"
Sep 08 15:30:07.436372 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:07.436345    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:30:07.438127 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:30:07.438099    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938315a2_b1e8_4e7f_98bd_0d615d857c5a.slice/crio-2b0a33ebc56d9046dda68fca9326d87a23e2228d9a46fb14fd618f9a0f2d5e78 WatchSource:0}: Error finding container 2b0a33ebc56d9046dda68fca9326d87a23e2228d9a46fb14fd618f9a0f2d5e78: Status 404 returned error can't find the container with id 2b0a33ebc56d9046dda68fca9326d87a23e2228d9a46fb14fd618f9a0f2d5e78
Sep 08 15:30:08.198340 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:08.198301    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" event={"ID":"938315a2-b1e8-4e7f-98bd-0d615d857c5a","Type":"ContainerStarted","Data":"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"}
Sep 08 15:30:08.198812 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:08.198382    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" event={"ID":"938315a2-b1e8-4e7f-98bd-0d615d857c5a","Type":"ContainerStarted","Data":"2b0a33ebc56d9046dda68fca9326d87a23e2228d9a46fb14fd618f9a0f2d5e78"}
Sep 08 15:30:08.198812 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:08.198404    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:08.216567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:08.216516    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" podStartSLOduration=2.216503646 podStartE2EDuration="2.216503646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:08.214096503 +0000 UTC m=+1050.056893725" watchObservedRunningTime="2026-09-08 15:30:08.216503646 +0000 UTC m=+1050.059300887"
Sep 08 15:30:08.306232 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:08.306202    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:30:13.204880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:13.204849    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:30:19.206134 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:19.206102    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:30:24.483945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.483907    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:30:24.487739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.487715    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.490458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.490436    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-020a18-dockercfg-8tr5v\""
Sep 08 15:30:24.490934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.490919    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-020a18\""
Sep 08 15:30:24.500348 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.500328    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:30:24.515245 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.515221    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txc4l\" (UniqueName: \"kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.515363 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.515283    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.616720 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.616690    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.616902 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.616757    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txc4l\" (UniqueName: \"kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.617445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.617425    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.628666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.628639    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txc4l\" (UniqueName: \"kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l\") pod \"payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:24.797489 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:24.797397    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:25.073752 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:25.073728    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:30:25.076487 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:30:25.076457    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fc746a_1eed_4958_b04d_2e826855e606.slice/crio-322f3626da7d90d32df7b974273767ac85a7f53a7383ee8063f6140a2137f13f WatchSource:0}: Error finding container 322f3626da7d90d32df7b974273767ac85a7f53a7383ee8063f6140a2137f13f: Status 404 returned error can't find the container with id 322f3626da7d90d32df7b974273767ac85a7f53a7383ee8063f6140a2137f13f
Sep 08 15:30:25.291831 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:25.291792    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" event={"ID":"67fc746a-1eed-4958-b04d-2e826855e606","Type":"ContainerStarted","Data":"25d3c423deb06bb022f072f206af3cda4e51a51b532ff2e0416fa9e4646efb95"}
Sep 08 15:30:25.292047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:25.291849    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" event={"ID":"67fc746a-1eed-4958-b04d-2e826855e606","Type":"ContainerStarted","Data":"322f3626da7d90d32df7b974273767ac85a7f53a7383ee8063f6140a2137f13f"}
Sep 08 15:30:25.292047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:25.291879    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:25.315515 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:25.315458    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" podStartSLOduration=1.315441963 podStartE2EDuration="1.315441963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:25.314376983 +0000 UTC m=+1067.157174207" watchObservedRunningTime="2026-09-08 15:30:25.315441963 +0000 UTC m=+1067.158239186"
Sep 08 15:30:29.020192 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:29.020157    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:30:35.631585 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.631505    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:30:35.635139 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.635123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.638919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.638900    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-565cd7\""
Sep 08 15:30:35.639112 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.639095    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-565cd7-dockercfg-2wnh4\""
Sep 08 15:30:35.646524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.646496    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:30:35.709259 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.709227    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cwd\" (UniqueName: \"kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.709427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.709282    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.810533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.810501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cwd\" (UniqueName: \"kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.810668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.810559    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.811394 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.811363    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.820228 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.820189    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cwd\" (UniqueName: \"kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd\") pod \"payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:35.945817 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:35.945747    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:36.219345 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.219321    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:30:36.221109 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:30:36.221085    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b7d815_292a_4009_9de2_59c7113ad9b1.slice/crio-2c481dbcba19bc922c042fb91c0afd00cbdb6f9b1732362938d492a6f5111c6b WatchSource:0}: Error finding container 2c481dbcba19bc922c042fb91c0afd00cbdb6f9b1732362938d492a6f5111c6b: Status 404 returned error can't find the container with id 2c481dbcba19bc922c042fb91c0afd00cbdb6f9b1732362938d492a6f5111c6b
Sep 08 15:30:36.300034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.300013    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:30:36.360317 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.360257    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" event={"ID":"30b7d815-292a-4009-9de2-59c7113ad9b1","Type":"ContainerStarted","Data":"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"}
Sep 08 15:30:36.360457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.360346    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" event={"ID":"30b7d815-292a-4009-9de2-59c7113ad9b1","Type":"ContainerStarted","Data":"2c481dbcba19bc922c042fb91c0afd00cbdb6f9b1732362938d492a6f5111c6b"}
Sep 08 15:30:36.360457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.360386    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:36.384178 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:36.384134    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" podStartSLOduration=1.38411733 podStartE2EDuration="1.38411733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:36.381660512 +0000 UTC m=+1078.224457735" watchObservedRunningTime="2026-09-08 15:30:36.38411733 +0000 UTC m=+1078.226914554"
Sep 08 15:30:39.649574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:39.649528    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:30:47.369225 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:47.369194    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:30:53.078480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.078444    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:30:53.083501 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.083482    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.087925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.087903    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d62b4e\""
Sep 08 15:30:53.103333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.103263    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:30:53.163895 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.163867    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.164034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.163913    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f4t\" (UniqueName: \"kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.264664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.264631    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.264802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.264686    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78f4t\" (UniqueName: \"kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.265439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.265422    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.276132 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.276112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f4t\" (UniqueName: \"kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t\") pod \"payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.393661 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.393639    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:53.670441 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:53.670414    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:30:53.671724 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:30:53.671700    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0bb4ec_7bfe_48e8_bb4a_29049c343fa3.slice/crio-3b223c3b812db8f7d8274949f947a19f5275265b7388fe9a1117d7770309fd14 WatchSource:0}: Error finding container 3b223c3b812db8f7d8274949f947a19f5275265b7388fe9a1117d7770309fd14: Status 404 returned error can't find the container with id 3b223c3b812db8f7d8274949f947a19f5275265b7388fe9a1117d7770309fd14
Sep 08 15:30:54.459122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:54.459088    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" event={"ID":"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3","Type":"ContainerStarted","Data":"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"}
Sep 08 15:30:54.459525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:54.459140    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" event={"ID":"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3","Type":"ContainerStarted","Data":"3b223c3b812db8f7d8274949f947a19f5275265b7388fe9a1117d7770309fd14"}
Sep 08 15:30:54.459525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:54.459299    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:30:54.478251 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:54.478207    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" podStartSLOduration=1.4781922060000001 podStartE2EDuration="1.478192206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:54.476897764 +0000 UTC m=+1096.319694986" watchObservedRunningTime="2026-09-08 15:30:54.478192206 +0000 UTC m=+1096.320989454"
Sep 08 15:30:57.937531 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:30:57.937488    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:31:05.466823 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:05.466788    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:31:14.331736 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.331701    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:31:14.335463 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.335440    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.338598 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.338575    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"auth-test-model-fc867c-kserve-self-signed-certs\""
Sep 08 15:31:14.338737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.338627    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"kube-root-ca.crt\""
Sep 08 15:31:14.338737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.338669    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"openshift-service-ca.crt\""
Sep 08 15:31:14.338737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.338633    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"default-dockercfg-w6q7z\""
Sep 08 15:31:14.352878 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.352855    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:31:14.359693 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359670    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.359816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359710    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.359883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359830    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.359883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359878    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.359987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359904    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xnqc\" (UniqueName: \"kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.359987 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.359937    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460621 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460576    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460633    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460667    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460755    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460789    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.460813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.460804    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xnqc\" (UniqueName: \"kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.461411 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.461383    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.461533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.461407    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.461533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.461503    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.463708 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.463681    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.465120 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.465094    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.470967 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.470942    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xnqc\" (UniqueName: \"kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc\") pod \"auth-test-model-fc867c-kserve-8457c46c95-wpxc6\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.646155 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.646123    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:14.937955 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:14.937928    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:31:14.940086 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:14.940056    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22e2f27_ee08_487b_a53c_5be2cdb5310c.slice/crio-fc9aa8a70c302447a3f3d2f619465af37d3eabe92566b8d4846e18684a1f143e WatchSource:0}: Error finding container fc9aa8a70c302447a3f3d2f619465af37d3eabe92566b8d4846e18684a1f143e: Status 404 returned error can't find the container with id fc9aa8a70c302447a3f3d2f619465af37d3eabe92566b8d4846e18684a1f143e
Sep 08 15:31:15.576081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:15.576045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" event={"ID":"a22e2f27-ee08-487b-a53c-5be2cdb5310c","Type":"ContainerStarted","Data":"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"}
Sep 08 15:31:15.576081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:15.576088    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" event={"ID":"a22e2f27-ee08-487b-a53c-5be2cdb5310c","Type":"ContainerStarted","Data":"fc9aa8a70c302447a3f3d2f619465af37d3eabe92566b8d4846e18684a1f143e"}
Sep 08 15:31:15.606308 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:15.606253    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" podStartSLOduration=1.6062379020000002 podStartE2EDuration="1.606237902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:15.604613237 +0000 UTC m=+1117.447410464" watchObservedRunningTime="2026-09-08 15:31:15.606237902 +0000 UTC m=+1117.449035124"
Sep 08 15:31:16.709601 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.709571    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:16.712899 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.712882    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:16.715953 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.715923    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8vmw2\""
Sep 08 15:31:16.729884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.729857    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:16.780183 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.780147    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssw7\" (UniqueName: \"kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7\") pod \"test-curl-7766-140604484658496\" (UID: \"c3462587-a75e-432a-a6f8-a0f680fb6e95\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:16.880848 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.880802    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssw7\" (UniqueName: \"kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7\") pod \"test-curl-7766-140604484658496\" (UID: \"c3462587-a75e-432a-a6f8-a0f680fb6e95\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:16.892274 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:16.892250    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssw7\" (UniqueName: \"kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7\") pod \"test-curl-7766-140604484658496\" (UID: \"c3462587-a75e-432a-a6f8-a0f680fb6e95\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:17.022660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:17.022584    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:17.300090 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:17.300063    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:17.302455 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:17.302426    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3462587_a75e_432a_a6f8_a0f680fb6e95.slice/crio-73871ad4b51d2b3176e31c9a5db4f756c2585376d8373ab21102a7ede7e74624 WatchSource:0}: Error finding container 73871ad4b51d2b3176e31c9a5db4f756c2585376d8373ab21102a7ede7e74624: Status 404 returned error can't find the container with id 73871ad4b51d2b3176e31c9a5db4f756c2585376d8373ab21102a7ede7e74624
Sep 08 15:31:17.588976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:17.588894    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484658496" event={"ID":"c3462587-a75e-432a-a6f8-a0f680fb6e95","Type":"ContainerStarted","Data":"73871ad4b51d2b3176e31c9a5db4f756c2585376d8373ab21102a7ede7e74624"}
Sep 08 15:31:18.596457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:18.596420    2568 generic.go:356] "Generic (PLEG): container finished" podID="c3462587-a75e-432a-a6f8-a0f680fb6e95" containerID="a420e18bd1d660be4f96b77cad11295c17d992a396bb6c036d3aa70f33409afe" exitCode=0
Sep 08 15:31:18.596457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:18.596463    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484658496" event={"ID":"c3462587-a75e-432a-a6f8-a0f680fb6e95","Type":"ContainerDied","Data":"a420e18bd1d660be4f96b77cad11295c17d992a396bb6c036d3aa70f33409afe"}
Sep 08 15:31:19.732737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.732710    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:19.803397 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.803361    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nssw7\" (UniqueName: \"kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7\") pod \"c3462587-a75e-432a-a6f8-a0f680fb6e95\" (UID: \"c3462587-a75e-432a-a6f8-a0f680fb6e95\") "
Sep 08 15:31:19.805464 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.805429    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7" (OuterVolumeSpecName: "kube-api-access-nssw7") pod "c3462587-a75e-432a-a6f8-a0f680fb6e95" (UID: "c3462587-a75e-432a-a6f8-a0f680fb6e95"). InnerVolumeSpecName "kube-api-access-nssw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:19.820232 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.820214    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7766-140604484658496_c3462587-a75e-432a-a6f8-a0f680fb6e95/test-curl-7766-140604484658496/0.log"
Sep 08 15:31:19.879751 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.879724    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:19.884456 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.884435    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:19.904565 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:19.904541    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nssw7\" (UniqueName: \"kubernetes.io/projected/c3462587-a75e-432a-a6f8-a0f680fb6e95-kube-api-access-nssw7\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:31:20.020745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.020710    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:20.021097 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.021084    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3462587-a75e-432a-a6f8-a0f680fb6e95" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:20.021157 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.021099    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3462587-a75e-432a-a6f8-a0f680fb6e95" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:20.021195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.021170    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3462587-a75e-432a-a6f8-a0f680fb6e95" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:20.024240 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.024222    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.035371 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.035344    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:20.107139 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.107094    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4zb\" (UniqueName: \"kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb\") pod \"test-curl-7766-140604484658496\" (UID: \"10b996d0-0646-4607-81ba-63cbfbea731d\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.208380 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.208268    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4zb\" (UniqueName: \"kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb\") pod \"test-curl-7766-140604484658496\" (UID: \"10b996d0-0646-4607-81ba-63cbfbea731d\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.222159 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.222122    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4zb\" (UniqueName: \"kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb\") pod \"test-curl-7766-140604484658496\" (UID: \"10b996d0-0646-4607-81ba-63cbfbea731d\") " pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.334394 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.334364    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.609799 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.609719    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:20.609799 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.609752    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73871ad4b51d2b3176e31c9a5db4f756c2585376d8373ab21102a7ede7e74624"
Sep 08 15:31:20.617828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.617806    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:20.619243 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:20.619217    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10b996d0_0646_4607_81ba_63cbfbea731d.slice/crio-9ab92b7b540e7aded10de9d03ade0068aca54ec5ff0203f3ad447b607c74d9c7 WatchSource:0}: Error finding container 9ab92b7b540e7aded10de9d03ade0068aca54ec5ff0203f3ad447b607c74d9c7: Status 404 returned error can't find the container with id 9ab92b7b540e7aded10de9d03ade0068aca54ec5ff0203f3ad447b607c74d9c7
Sep 08 15:31:20.624176 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.624151    2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7766-140604484658496" oldPodUID="c3462587-a75e-432a-a6f8-a0f680fb6e95" podUID="10b996d0-0646-4607-81ba-63cbfbea731d"
Sep 08 15:31:20.642765 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.642736    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:31:20.676985 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:20.676956    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3462587-a75e-432a-a6f8-a0f680fb6e95" path="/var/lib/kubelet/pods/c3462587-a75e-432a-a6f8-a0f680fb6e95/volumes"
Sep 08 15:31:21.624527 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:21.624481    2568 generic.go:356] "Generic (PLEG): container finished" podID="10b996d0-0646-4607-81ba-63cbfbea731d" containerID="6577c332d42f0e3dd3a7d0e62f187ff1da7aefaa00edcf6be7c5f731f5a1902e" exitCode=0
Sep 08 15:31:21.624988 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:21.624545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484658496" event={"ID":"10b996d0-0646-4607-81ba-63cbfbea731d","Type":"ContainerDied","Data":"6577c332d42f0e3dd3a7d0e62f187ff1da7aefaa00edcf6be7c5f731f5a1902e"}
Sep 08 15:31:21.624988 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:21.624589    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484658496" event={"ID":"10b996d0-0646-4607-81ba-63cbfbea731d","Type":"ContainerStarted","Data":"9ab92b7b540e7aded10de9d03ade0068aca54ec5ff0203f3ad447b607c74d9c7"}
Sep 08 15:31:22.759603 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.759580    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:22.794857 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.794829    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7766-140604484658496_10b996d0-0646-4607-81ba-63cbfbea731d/test-curl-7766-140604484658496/0.log"
Sep 08 15:31:22.833310 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.833268    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4zb\" (UniqueName: \"kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb\") pod \"10b996d0-0646-4607-81ba-63cbfbea731d\" (UID: \"10b996d0-0646-4607-81ba-63cbfbea731d\") "
Sep 08 15:31:22.835484 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.835453    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb" (OuterVolumeSpecName: "kube-api-access-lj4zb") pod "10b996d0-0646-4607-81ba-63cbfbea731d" (UID: "10b996d0-0646-4607-81ba-63cbfbea731d"). InnerVolumeSpecName "kube-api-access-lj4zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:22.840528 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.840502    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:22.842983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.842951    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7766-140604484658496"]
Sep 08 15:31:22.934524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.934456    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4zb\" (UniqueName: \"kubernetes.io/projected/10b996d0-0646-4607-81ba-63cbfbea731d-kube-api-access-lj4zb\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:31:22.980056 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.980027    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:22.980413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.980399    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:22.980464 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.980414    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:22.980498 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.980489    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" containerName="test-curl-7766-140604484658496"
Sep 08 15:31:22.983480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.983462    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:22.997557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:22.997531    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:23.035455 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.035425    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcg4z\" (UniqueName: \"kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z\") pod \"test-curl-7766-140604484659072\" (UID: \"62ac8392-790f-47c9-af10-7609ae6bbede\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:23.136679 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.136647    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcg4z\" (UniqueName: \"kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z\") pod \"test-curl-7766-140604484659072\" (UID: \"62ac8392-790f-47c9-af10-7609ae6bbede\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:23.157406 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.157375    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcg4z\" (UniqueName: \"kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z\") pod \"test-curl-7766-140604484659072\" (UID: \"62ac8392-790f-47c9-af10-7609ae6bbede\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:23.292479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.292408    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:23.589482 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.589452    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:23.589621 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:23.589599    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ac8392_790f_47c9_af10_7609ae6bbede.slice/crio-302c169619764bc9fbfaa9e306b9d8fe62be87dbd2e5960e57885f3448324093 WatchSource:0}: Error finding container 302c169619764bc9fbfaa9e306b9d8fe62be87dbd2e5960e57885f3448324093: Status 404 returned error can't find the container with id 302c169619764bc9fbfaa9e306b9d8fe62be87dbd2e5960e57885f3448324093
Sep 08 15:31:23.635976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.635951    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:23.636099 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.635999    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab92b7b540e7aded10de9d03ade0068aca54ec5ff0203f3ad447b607c74d9c7"
Sep 08 15:31:23.637348 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:23.637313    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484659072" event={"ID":"62ac8392-790f-47c9-af10-7609ae6bbede","Type":"ContainerStarted","Data":"302c169619764bc9fbfaa9e306b9d8fe62be87dbd2e5960e57885f3448324093"}
Sep 08 15:31:23.647060 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:31:23.647033    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7766-140604484658496\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:24.643905 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.643870    2568 generic.go:356] "Generic (PLEG): container finished" podID="62ac8392-790f-47c9-af10-7609ae6bbede" containerID="e0c32ec9590a2b877737aa5e5f98a2a2632954eb2347259b492b8fa1f7a785a9" exitCode=0
Sep 08 15:31:24.644364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.643950    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484659072" event={"ID":"62ac8392-790f-47c9-af10-7609ae6bbede","Type":"ContainerDied","Data":"e0c32ec9590a2b877737aa5e5f98a2a2632954eb2347259b492b8fa1f7a785a9"}
Sep 08 15:31:24.646996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.646978    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:24.647115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.647004    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:24.659324 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.659284    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:24.675149 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:24.675127    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" path="/var/lib/kubelet/pods/10b996d0-0646-4607-81ba-63cbfbea731d/volumes"
Sep 08 15:31:24.690671 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:31:24.690644    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7766-140604484658496\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="10b996d0-0646-4607-81ba-63cbfbea731d" pod="opendatahub/test-curl-7766-140604484658496"
Sep 08 15:31:25.661333 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.661285    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:31:25.785588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.785557    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:25.826242 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.826210    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7766-140604484659072_62ac8392-790f-47c9-af10-7609ae6bbede/test-curl-7766-140604484659072/0.log"
Sep 08 15:31:25.861108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.861084    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcg4z\" (UniqueName: \"kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z\") pod \"62ac8392-790f-47c9-af10-7609ae6bbede\" (UID: \"62ac8392-790f-47c9-af10-7609ae6bbede\") "
Sep 08 15:31:25.862346 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.862322    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:25.863017 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.862988    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z" (OuterVolumeSpecName: "kube-api-access-qcg4z") pod "62ac8392-790f-47c9-af10-7609ae6bbede" (UID: "62ac8392-790f-47c9-af10-7609ae6bbede"). InnerVolumeSpecName "kube-api-access-qcg4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:25.867683 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.867658    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:25.961996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:25.961923    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcg4z\" (UniqueName: \"kubernetes.io/projected/62ac8392-790f-47c9-af10-7609ae6bbede-kube-api-access-qcg4z\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:31:26.017809 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.017777    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:26.018225 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.018206    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ac8392-790f-47c9-af10-7609ae6bbede" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:26.018338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.018227    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ac8392-790f-47c9-af10-7609ae6bbede" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:26.018405 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.018345    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ac8392-790f-47c9-af10-7609ae6bbede" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:26.021400 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.021372    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.056862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.056840    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:26.062343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.062323    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfx4\" (UniqueName: \"kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4\") pod \"test-curl-7766-140604484659072\" (UID: \"1e23dc52-76d7-4a02-935f-68fba22116e2\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.163001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.162968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfx4\" (UniqueName: \"kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4\") pod \"test-curl-7766-140604484659072\" (UID: \"1e23dc52-76d7-4a02-935f-68fba22116e2\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.174771 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.174739    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfx4\" (UniqueName: \"kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4\") pod \"test-curl-7766-140604484659072\" (UID: \"1e23dc52-76d7-4a02-935f-68fba22116e2\") " pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.330828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.330758    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.615116 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.615089    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:26.615274 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:26.615114    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e23dc52_76d7_4a02_935f_68fba22116e2.slice/crio-176cbb8eb7d813459d87a38fab6dc16ef935b6d522b8959322f106f6f5ae25f0 WatchSource:0}: Error finding container 176cbb8eb7d813459d87a38fab6dc16ef935b6d522b8959322f106f6f5ae25f0: Status 404 returned error can't find the container with id 176cbb8eb7d813459d87a38fab6dc16ef935b6d522b8959322f106f6f5ae25f0
Sep 08 15:31:26.656373 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.656347    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:26.656373 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.656363    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="302c169619764bc9fbfaa9e306b9d8fe62be87dbd2e5960e57885f3448324093"
Sep 08 15:31:26.657668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.657634    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484659072" event={"ID":"1e23dc52-76d7-4a02-935f-68fba22116e2","Type":"ContainerStarted","Data":"176cbb8eb7d813459d87a38fab6dc16ef935b6d522b8959322f106f6f5ae25f0"}
Sep 08 15:31:26.670893 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.670866    2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7766-140604484659072" oldPodUID="62ac8392-790f-47c9-af10-7609ae6bbede" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2"
Sep 08 15:31:26.676685 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:26.676661    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ac8392-790f-47c9-af10-7609ae6bbede" path="/var/lib/kubelet/pods/62ac8392-790f-47c9-af10-7609ae6bbede/volumes"
Sep 08 15:31:27.666232 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:27.666199    2568 generic.go:356] "Generic (PLEG): container finished" podID="1e23dc52-76d7-4a02-935f-68fba22116e2" containerID="b3558b6778f584333b49da0bad95f6a4e1ac23052040879da0dfcc8171aefd5d" exitCode=0
Sep 08 15:31:27.666453 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:27.666288    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7766-140604484659072" event={"ID":"1e23dc52-76d7-4a02-935f-68fba22116e2","Type":"ContainerDied","Data":"b3558b6778f584333b49da0bad95f6a4e1ac23052040879da0dfcc8171aefd5d"}
Sep 08 15:31:28.806279 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.806257    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:28.844796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.844763    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7766-140604484659072_1e23dc52-76d7-4a02-935f-68fba22116e2/test-curl-7766-140604484659072/0.log"
Sep 08 15:31:28.881015 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.880983    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:28.888654 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.888631    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfx4\" (UniqueName: \"kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4\") pod \"1e23dc52-76d7-4a02-935f-68fba22116e2\" (UID: \"1e23dc52-76d7-4a02-935f-68fba22116e2\") "
Sep 08 15:31:28.890877 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.890851    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4" (OuterVolumeSpecName: "kube-api-access-tlfx4") pod "1e23dc52-76d7-4a02-935f-68fba22116e2" (UID: "1e23dc52-76d7-4a02-935f-68fba22116e2"). InnerVolumeSpecName "kube-api-access-tlfx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:31:28.892813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.892787    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7766-140604484659072"]
Sep 08 15:31:28.989747 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:28.989663    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfx4\" (UniqueName: \"kubernetes.io/projected/1e23dc52-76d7-4a02-935f-68fba22116e2-kube-api-access-tlfx4\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:31:29.680229 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:29.680198    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:29.680398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:29.680206    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176cbb8eb7d813459d87a38fab6dc16ef935b6d522b8959322f106f6f5ae25f0"
Sep 08 15:31:29.693152 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:31:29.693129    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7766-140604484659072\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2" pod="opendatahub/test-curl-7766-140604484659072"
Sep 08 15:31:30.681027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:30.680991    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2" path="/var/lib/kubelet/pods/1e23dc52-76d7-4a02-935f-68fba22116e2/volumes"
Sep 08 15:31:38.999870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:38.999836    2568 scope.go:117] "RemoveContainer" containerID="4f35679f993da399c3f8c62fe87982b2199d65246e277137ec198c2c55b093ff"
Sep 08 15:31:39.008509 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.008487    2568 scope.go:117] "RemoveContainer" containerID="3a2c83991ac65e702e31f69402bd4cca2c9efe23a9cb73a32512e6ea56479a30"
Sep 08 15:31:39.551569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.551534    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:31:39.551911 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.551897    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:39.551964 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.551913    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:39.552001 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.551988    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e23dc52-76d7-4a02-935f-68fba22116e2" containerName="test-curl-7766-140604484659072"
Sep 08 15:31:39.556469 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.556452    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.560661 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.560639    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"default-dockercfg-gllms\""
Sep 08 15:31:39.561405 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.561387    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"openshift-service-ca.crt\""
Sep 08 15:31:39.561405 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.561399    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"kube-root-ca.crt\""
Sep 08 15:31:39.561555 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.561539    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"auth-test-model-565cd7-kserve-self-signed-certs\""
Sep 08 15:31:39.569748 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.569724    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:31:39.679993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.679961    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.679993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.679994    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.680226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.680021    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.680226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.680131    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966hm\" (UniqueName: \"kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.680226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.680181    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.680226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.680200    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.780634 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780604    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966hm\" (UniqueName: \"kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.780835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780659    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.780835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780770    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.780835 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780808    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.781044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780837    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.781044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.780883    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.781193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.781168    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.781193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.781177    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.781346 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.781276    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.782911 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.782890    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.783190 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.783171    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.790481 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.790455    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966hm\" (UniqueName: \"kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm\") pod \"auth-test-model-565cd7-kserve-c79d44b99-4xz29\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:39.866156 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:39.866080    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:40.150122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:40.150094    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:31:40.151910 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:31:40.151881    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8242aa3d_da1b_4da4_b571_51ef092e1f92.slice/crio-1511c09eafd17df761cbc31647214e9f97e71c901dcadd31479902148e27bb7b WatchSource:0}: Error finding container 1511c09eafd17df761cbc31647214e9f97e71c901dcadd31479902148e27bb7b: Status 404 returned error can't find the container with id 1511c09eafd17df761cbc31647214e9f97e71c901dcadd31479902148e27bb7b
Sep 08 15:31:40.750711 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:40.750675    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" event={"ID":"8242aa3d-da1b-4da4-b571-51ef092e1f92","Type":"ContainerStarted","Data":"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"}
Sep 08 15:31:40.750886 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:40.750723    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" event={"ID":"8242aa3d-da1b-4da4-b571-51ef092e1f92","Type":"ContainerStarted","Data":"1511c09eafd17df761cbc31647214e9f97e71c901dcadd31479902148e27bb7b"}
Sep 08 15:31:40.778865 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:40.778816    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" podStartSLOduration=1.77880236 podStartE2EDuration="1.77880236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:40.778393398 +0000 UTC m=+1142.621190624" watchObservedRunningTime="2026-09-08 15:31:40.77880236 +0000 UTC m=+1142.621599582"
Sep 08 15:31:43.008023 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:43.007981    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:31:49.866737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:49.866706    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:49.867130 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:49.866757    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:49.879169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:49.879145    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:31:50.819913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:31:50.819881    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:32:08.711260 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:08.711187    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:32:20.816163 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.816129    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:32:20.819876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.819853    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.822712 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.822687    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-26dbe7df\"/\"openshift-service-ca.crt\""
Sep 08 15:32:20.822860 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.822742    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-26dbe7df\"/\"kube-root-ca.crt\""
Sep 08 15:32:20.822860 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.822688    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-26dbe7df\"/\"e2e-lifecycle-model-26dbe7df-kserve-self-signed-certs\""
Sep 08 15:32:20.822860 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.822687    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-26dbe7df\"/\"default-dockercfg-tcqdm\""
Sep 08 15:32:20.831421 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.831397    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:32:20.925971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.925938    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.925971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.925972    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.926220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.926009    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.926220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.926026    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.926220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.926053    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:20.926220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:20.926094    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgrh\" (UniqueName: \"kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.026891 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.026856    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgrh\" (UniqueName: \"kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.026908    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.026929    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.026968    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.026983    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.027001    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027449 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.027425    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.027459    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.027525 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.027480    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.029118 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.029100    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.029597 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.029575    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.035460 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.035440    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgrh\" (UniqueName: \"kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh\") pod \"e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") " pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.131181 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.131159    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:21.405804 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.405762    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:32:21.407277 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:32:21.407251    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda339b29d_ad3d_4a4b_8e39_8dcc1f14f139.slice/crio-297fb47513b499f5f21c355d43ce03311803c99f349fdd4cfacf941c3e011734 WatchSource:0}: Error finding container 297fb47513b499f5f21c355d43ce03311803c99f349fdd4cfacf941c3e011734: Status 404 returned error can't find the container with id 297fb47513b499f5f21c355d43ce03311803c99f349fdd4cfacf941c3e011734
Sep 08 15:32:21.983524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.983483    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" event={"ID":"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139","Type":"ContainerStarted","Data":"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"}
Sep 08 15:32:21.983524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:21.983524    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" event={"ID":"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139","Type":"ContainerStarted","Data":"297fb47513b499f5f21c355d43ce03311803c99f349fdd4cfacf941c3e011734"}
Sep 08 15:32:22.000821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:22.000773    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" podStartSLOduration=2.000758306 podStartE2EDuration="2.000758306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:32:21.99944302 +0000 UTC m=+1183.842240254" watchObservedRunningTime="2026-09-08 15:32:22.000758306 +0000 UTC m=+1183.843555530"
Sep 08 15:32:26.005494 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:26.005460    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:32:31.132013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:31.131975    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:31.132523 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:31.132220    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:31.144720 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:31.144695    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:32.051115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:32.051081    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:32:38.688027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:38.687997    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:32:38.690160 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:38.690138    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:32:44.195139 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:32:44.195103    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:33:05.502218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:05.502184    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:33:23.800427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:23.800391    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:33:27.974431 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:27.974391    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:33:39.698797 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:39.698712    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:33:42.994128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:42.994085    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" podUID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" containerName="main" containerID="cri-o://0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42" gracePeriod=45
Sep 08 15:33:43.250739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.250679    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:33:43.347926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.347895    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.347926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.347929    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.348136 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.347950    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqgrh\" (UniqueName: \"kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.348136 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.347997    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.348136 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348024    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.348136 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348044    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache\") pod \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\" (UID: \"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139\") "
Sep 08 15:33:43.348361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348264    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home" (OuterVolumeSpecName: "home") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:43.348424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348389    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:43.348482 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348470    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.348538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.348487    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache" (OuterVolumeSpecName: "model-cache") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:43.350156 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.350129    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:33:43.350273 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.350255    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm" (OuterVolumeSpecName: "dshm") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:33:43.350398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.350378    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh" (OuterVolumeSpecName: "kube-api-access-bqgrh") pod "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" (UID: "a339b29d-ad3d-4a4b-8e39-8dcc1f14f139"). InnerVolumeSpecName "kube-api-access-bqgrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:33:43.447415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.447382    2568 generic.go:356] "Generic (PLEG): container finished" podID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" containerID="0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42" exitCode=0
Sep 08 15:33:43.447580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.447425    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"
Sep 08 15:33:43.447580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.447444    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" event={"ID":"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139","Type":"ContainerDied","Data":"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"}
Sep 08 15:33:43.447580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.447476    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n" event={"ID":"a339b29d-ad3d-4a4b-8e39-8dcc1f14f139","Type":"ContainerDied","Data":"297fb47513b499f5f21c355d43ce03311803c99f349fdd4cfacf941c3e011734"}
Sep 08 15:33:43.447580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.447491    2568 scope.go:117] "RemoveContainer" containerID="0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"
Sep 08 15:33:43.455524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.449757    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.455524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.449793    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgrh\" (UniqueName: \"kubernetes.io/projected/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-kube-api-access-bqgrh\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.455524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.449812    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.455524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.449841    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.455524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.449861    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:33:43.462698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.462679    2568 scope.go:117] "RemoveContainer" containerID="0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"
Sep 08 15:33:43.463017 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:33:43.462997    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42\": container with ID starting with 0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42 not found: ID does not exist" containerID="0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"
Sep 08 15:33:43.463081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.463024    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42"} err="failed to get container status \"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42\": rpc error: code = NotFound desc = could not find container \"0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42\": container with ID starting with 0562110910efbce1b7189a80da75ae85052a1e971105cc37a8d598b5507c7d42 not found: ID does not exist"
Sep 08 15:33:43.472440 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.472416    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:33:43.477998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:43.477978    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-26dbe7df/e2e-lifecycle-model-26dbe7df-kserve-77cc876bb7-q9l5n"]
Sep 08 15:33:44.677581 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:44.677544    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" path="/var/lib/kubelet/pods/a339b29d-ad3d-4a4b-8e39-8dcc1f14f139/volumes"
Sep 08 15:33:47.641763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:47.641729    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:33:48.603980 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:48.603943    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:33:54.195794 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:33:54.195758    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:34:02.662564 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:02.662505    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" podUID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" containerName="main" containerID="cri-o://a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f" gracePeriod=45
Sep 08 15:34:02.909703 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:02.909676    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:34:03.013465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013373    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013409    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013456    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013497    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xnqc\" (UniqueName: \"kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013539    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013564    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir\") pod \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\" (UID: \"a22e2f27-ee08-487b-a53c-5be2cdb5310c\") "
Sep 08 15:34:03.013753 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013702    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.013923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013811    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home" (OuterVolumeSpecName: "home") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.013923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013841    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.013923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.013899    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.015883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.015853    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:03.015883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.015865    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm" (OuterVolumeSpecName: "dshm") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.016000 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.015924    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc" (OuterVolumeSpecName: "kube-api-access-5xnqc") pod "a22e2f27-ee08-487b-a53c-5be2cdb5310c" (UID: "a22e2f27-ee08-487b-a53c-5be2cdb5310c"). InnerVolumeSpecName "kube-api-access-5xnqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:03.114331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.114274    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xnqc\" (UniqueName: \"kubernetes.io/projected/a22e2f27-ee08-487b-a53c-5be2cdb5310c-kube-api-access-5xnqc\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.114331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.114326    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.114331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.114338    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.114573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.114345    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.114573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.114353    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22e2f27-ee08-487b-a53c-5be2cdb5310c-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.341444 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341368    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:34:03.341795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341763    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" containerName="main"
Sep 08 15:34:03.341795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341778    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" containerName="main"
Sep 08 15:34:03.341795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341793    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" containerName="main"
Sep 08 15:34:03.341795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341798    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" containerName="main"
Sep 08 15:34:03.342060 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341867    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a339b29d-ad3d-4a4b-8e39-8dcc1f14f139" containerName="main"
Sep 08 15:34:03.342060 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.341878    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" containerName="main"
Sep 08 15:34:03.345085 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.345067    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.348046 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.348024    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"sub-test-model-fc867c-kserve-self-signed-certs\""
Sep 08 15:34:03.358726 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.358701    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:34:03.416875 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.416840    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.416875 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.416878    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.417094 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.416903    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzf57\" (UniqueName: \"kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.417094 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.417011    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.417094 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.417043    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.417094 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.417070    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518357 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518321    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518562 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518375    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518562 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518414    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518562 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518441    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzf57\" (UniqueName: \"kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518562 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518496    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518562 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518522    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518863 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518839    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518863    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.518960 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.518943    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.520698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.520676    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.520970 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.520951    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.530769 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.530745    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzf57\" (UniqueName: \"kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57\") pod \"sub-test-model-fc867c-kserve-568695bbcc-6q98z\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.562739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.562710    2568 generic.go:356] "Generic (PLEG): container finished" podID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" containerID="a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f" exitCode=0
Sep 08 15:34:03.562871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.562764    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"
Sep 08 15:34:03.562871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.562794    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" event={"ID":"a22e2f27-ee08-487b-a53c-5be2cdb5310c","Type":"ContainerDied","Data":"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"}
Sep 08 15:34:03.562871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.562837    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6" event={"ID":"a22e2f27-ee08-487b-a53c-5be2cdb5310c","Type":"ContainerDied","Data":"fc9aa8a70c302447a3f3d2f619465af37d3eabe92566b8d4846e18684a1f143e"}
Sep 08 15:34:03.562871 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.562855    2568 scope.go:117] "RemoveContainer" containerID="a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"
Sep 08 15:34:03.572576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.572559    2568 scope.go:117] "RemoveContainer" containerID="a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"
Sep 08 15:34:03.572829 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:34:03.572812    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f\": container with ID starting with a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f not found: ID does not exist" containerID="a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"
Sep 08 15:34:03.572898 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.572839    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f"} err="failed to get container status \"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f\": rpc error: code = NotFound desc = could not find container \"a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f\": container with ID starting with a4256633b09c10fa960205bbc48075ed95971fd7d33049c3cf2e81f7dac45d7f not found: ID does not exist"
Sep 08 15:34:03.591171 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.591141    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:34:03.599524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.599469    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/auth-test-model-fc867c-kserve-8457c46c95-wpxc6"]
Sep 08 15:34:03.623758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.623730    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" podUID="8242aa3d-da1b-4da4-b571-51ef092e1f92" containerName="main" containerID="cri-o://02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421" gracePeriod=45
Sep 08 15:34:03.654261 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.654236    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:03.859140 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.859083    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:34:03.922662 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.922622    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.922862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.922761    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.922862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.922810    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.922862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.922845    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.923036 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.922901    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home" (OuterVolumeSpecName: "home") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.923098 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923032    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-966hm\" (UniqueName: \"kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.923098 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923025    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache" (OuterVolumeSpecName: "model-cache") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.923098 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923071    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm\") pod \"8242aa3d-da1b-4da4-b571-51ef092e1f92\" (UID: \"8242aa3d-da1b-4da4-b571-51ef092e1f92\") "
Sep 08 15:34:03.923098 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923088    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.923445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923424    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.923584 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923447    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.923584 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.923461    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:03.925465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.925386    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:34:03.925569 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.925553    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm" (OuterVolumeSpecName: "dshm") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:34:03.925628 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.925580    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm" (OuterVolumeSpecName: "kube-api-access-966hm") pod "8242aa3d-da1b-4da4-b571-51ef092e1f92" (UID: "8242aa3d-da1b-4da4-b571-51ef092e1f92"). InnerVolumeSpecName "kube-api-access-966hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:34:03.926312 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:03.926275    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:34:03.927888 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:34:03.927866    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd966e063_e13e_4f82_b001_d5546af3daaa.slice/crio-1773bcea9ce7e363865474c33e0088e5325c2f766b7f22c76702f5d437f3d449 WatchSource:0}: Error finding container 1773bcea9ce7e363865474c33e0088e5325c2f766b7f22c76702f5d437f3d449: Status 404 returned error can't find the container with id 1773bcea9ce7e363865474c33e0088e5325c2f766b7f22c76702f5d437f3d449
Sep 08 15:34:04.024936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.024899    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8242aa3d-da1b-4da4-b571-51ef092e1f92-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:04.024936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.024933    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966hm\" (UniqueName: \"kubernetes.io/projected/8242aa3d-da1b-4da4-b571-51ef092e1f92-kube-api-access-966hm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:04.025148 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.024947    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8242aa3d-da1b-4da4-b571-51ef092e1f92-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:34:04.570334 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.570271    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" event={"ID":"d966e063-e13e-4f82-b001-d5546af3daaa","Type":"ContainerStarted","Data":"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"}
Sep 08 15:34:04.570518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.570340    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" event={"ID":"d966e063-e13e-4f82-b001-d5546af3daaa","Type":"ContainerStarted","Data":"1773bcea9ce7e363865474c33e0088e5325c2f766b7f22c76702f5d437f3d449"}
Sep 08 15:34:04.571626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.571603    2568 generic.go:356] "Generic (PLEG): container finished" podID="8242aa3d-da1b-4da4-b571-51ef092e1f92" containerID="02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421" exitCode=0
Sep 08 15:34:04.571730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.571636    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"
Sep 08 15:34:04.571730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.571677    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" event={"ID":"8242aa3d-da1b-4da4-b571-51ef092e1f92","Type":"ContainerDied","Data":"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"}
Sep 08 15:34:04.571730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.571704    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29" event={"ID":"8242aa3d-da1b-4da4-b571-51ef092e1f92","Type":"ContainerDied","Data":"1511c09eafd17df761cbc31647214e9f97e71c901dcadd31479902148e27bb7b"}
Sep 08 15:34:04.571730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.571727    2568 scope.go:117] "RemoveContainer" containerID="02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"
Sep 08 15:34:04.581943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.581924    2568 scope.go:117] "RemoveContainer" containerID="02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"
Sep 08 15:34:04.582234 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:34:04.582215    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421\": container with ID starting with 02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421 not found: ID does not exist" containerID="02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"
Sep 08 15:34:04.582315 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.582241    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421"} err="failed to get container status \"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421\": rpc error: code = NotFound desc = could not find container \"02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421\": container with ID starting with 02312ae535252242a134f85cc10985e70ba3522e0c603e07e086d8c069141421 not found: ID does not exist"
Sep 08 15:34:04.589491 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.589448    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" podStartSLOduration=1.589432459 podStartE2EDuration="1.589432459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:04.587675143 +0000 UTC m=+1286.430472367" watchObservedRunningTime="2026-09-08 15:34:04.589432459 +0000 UTC m=+1286.432229731"
Sep 08 15:34:04.601693 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.601669    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:34:04.603785 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.603765    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/auth-test-model-565cd7-kserve-c79d44b99-4xz29"]
Sep 08 15:34:04.676529 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.676495    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8242aa3d-da1b-4da4-b571-51ef092e1f92" path="/var/lib/kubelet/pods/8242aa3d-da1b-4da4-b571-51ef092e1f92/volumes"
Sep 08 15:34:04.676925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:04.676912    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22e2f27-ee08-487b-a53c-5be2cdb5310c" path="/var/lib/kubelet/pods/a22e2f27-ee08-487b-a53c-5be2cdb5310c/volumes"
Sep 08 15:34:09.500945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:09.500907    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:34:13.654713 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:13.654682    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:13.655122 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:13.654747    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:13.667515 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:13.667481    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:14.643643 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:14.643608    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:34:20.533592 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:20.533544    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:34:33.199589 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:33.199551    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:34:52.327587 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.327556    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:34:52.328042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.327957    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8242aa3d-da1b-4da4-b571-51ef092e1f92" containerName="main"
Sep 08 15:34:52.328042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.327969    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8242aa3d-da1b-4da4-b571-51ef092e1f92" containerName="main"
Sep 08 15:34:52.328042 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.328039    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8242aa3d-da1b-4da4-b571-51ef092e1f92" containerName="main"
Sep 08 15:34:52.332378 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.332358    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.338388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.338364    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"sub-test-model-565cd7-kserve-self-signed-certs\""
Sep 08 15:34:52.338496 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.338410    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"default-dockercfg-gllms\""
Sep 08 15:34:52.338496 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.338463    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"kube-root-ca.crt\""
Sep 08 15:34:52.338735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.338718    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"openshift-service-ca.crt\""
Sep 08 15:34:52.345901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.345881    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:34:52.437307 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437272    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.437485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437338    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.437485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437399    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.437485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437453    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.437593 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437491    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.437593 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.437529    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggwz\" (UniqueName: \"kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538444    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggwz\" (UniqueName: \"kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538481    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538501    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538616    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538664    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.538967 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538944    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.539038 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538965    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.539038 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.538987    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.540925 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.540904    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.541189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.541171    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.548151 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.548132    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggwz\" (UniqueName: \"kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz\") pod \"sub-test-model-565cd7-kserve-56bbfdf979-bkt6c\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.641184 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.641160    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:34:52.927749 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:52.927725    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:34:52.929910 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:34:52.929881    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41f5a744_0f09_4e73_91a7_a4d4e74d3420.slice/crio-695f4b5a37473130747c6acc3aeb6b8729fe1ffb4172284b18fe9c891238f293 WatchSource:0}: Error finding container 695f4b5a37473130747c6acc3aeb6b8729fe1ffb4172284b18fe9c891238f293: Status 404 returned error can't find the container with id 695f4b5a37473130747c6acc3aeb6b8729fe1ffb4172284b18fe9c891238f293
Sep 08 15:34:53.851003 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:53.850964    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" event={"ID":"41f5a744-0f09-4e73-91a7-a4d4e74d3420","Type":"ContainerStarted","Data":"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"}
Sep 08 15:34:53.851398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:53.851009    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" event={"ID":"41f5a744-0f09-4e73-91a7-a4d4e74d3420","Type":"ContainerStarted","Data":"695f4b5a37473130747c6acc3aeb6b8729fe1ffb4172284b18fe9c891238f293"}
Sep 08 15:34:53.875146 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:53.875099    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" podStartSLOduration=1.8750845950000001 podStartE2EDuration="1.875084595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:34:53.874972352 +0000 UTC m=+1335.717769575" watchObservedRunningTime="2026-09-08 15:34:53.875084595 +0000 UTC m=+1335.717881820"
Sep 08 15:34:57.711927 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:34:57.711889    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:35:02.642230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:02.642144    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:35:02.642230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:02.642189    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:35:02.655260 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:02.655227    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:35:02.914639 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:02.914546    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:35:11.210583 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:11.210544    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:35:28.899837 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:28.899798    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:35:40.759543 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.759497    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:35:40.763226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.763203    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.765755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.765735    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0e34cd-serving-cert\""
Sep 08 15:35:40.779718 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.779693    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:35:40.864050 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.864013    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.864218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.864059    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzj8z\" (UniqueName: \"kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.864218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.864081    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.864218 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.864101    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.964850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.964808    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.965032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.964857    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzj8z\" (UniqueName: \"kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.965032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.964977    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.965032 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.965026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.965750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.965727    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.967427 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.967407    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.967567 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.967548    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:40.973540 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:40.973515    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzj8z\" (UniqueName: \"kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z\") pod \"maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:41.026735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.026673    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7769-6b9828"]
Sep 08 15:35:41.030241 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.030224    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:41.033264 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.033241    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8vmw2\""
Sep 08 15:35:41.040436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.040410    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7769-6b9828"]
Sep 08 15:35:41.065687 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.065655    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5bh\" (UniqueName: \"kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh\") pod \"mt-curl-7769-6b9828\" (UID: \"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb\") " pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:41.077454 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.077425    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:41.166809 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.166774    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5bh\" (UniqueName: \"kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh\") pod \"mt-curl-7769-6b9828\" (UID: \"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb\") " pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:41.175861 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.175831    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5bh\" (UniqueName: \"kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh\") pod \"mt-curl-7769-6b9828\" (UID: \"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb\") " pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:41.341323 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.341220    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:41.365594 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.365564    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:35:41.366307 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:35:41.366265    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f750439_2fe2_42cc_9e3a_4dfe9942e122.slice/crio-6c3b1c3d68c00eabd9a4c91a79db658b02b2d8a665122d522111e90185ded11d WatchSource:0}: Error finding container 6c3b1c3d68c00eabd9a4c91a79db658b02b2d8a665122d522111e90185ded11d: Status 404 returned error can't find the container with id 6c3b1c3d68c00eabd9a4c91a79db658b02b2d8a665122d522111e90185ded11d
Sep 08 15:35:41.367842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.367823    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:35:41.618546 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:41.618520    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7769-6b9828"]
Sep 08 15:35:41.619984 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:35:41.619956    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a681e81_9b10_456d_ba5f_bb4b6e9a66bb.slice/crio-94220d848690d8ab9f83c51c3642506674e2cf0f40d2498ecdaee2943722cc9b WatchSource:0}: Error finding container 94220d848690d8ab9f83c51c3642506674e2cf0f40d2498ecdaee2943722cc9b: Status 404 returned error can't find the container with id 94220d848690d8ab9f83c51c3642506674e2cf0f40d2498ecdaee2943722cc9b
Sep 08 15:35:42.128521 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:42.128484    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7769-6b9828" event={"ID":"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb","Type":"ContainerStarted","Data":"94220d848690d8ab9f83c51c3642506674e2cf0f40d2498ecdaee2943722cc9b"}
Sep 08 15:35:42.130350 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:42.130320    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" event={"ID":"4f750439-2fe2-42cc-9e3a-4dfe9942e122","Type":"ContainerStarted","Data":"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"}
Sep 08 15:35:42.130485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:42.130368    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" event={"ID":"4f750439-2fe2-42cc-9e3a-4dfe9942e122","Type":"ContainerStarted","Data":"6c3b1c3d68c00eabd9a4c91a79db658b02b2d8a665122d522111e90185ded11d"}
Sep 08 15:35:42.130485 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:42.130402    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:42.152876 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:42.152822    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" podStartSLOduration=1.8205158959999999 podStartE2EDuration="2.15280389s" totalImagesPullingTime="332.287994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:35:42.151062639 +0000 UTC m=+1383.993859863" watchObservedRunningTime="2026-09-08 15:35:42.15280389 +0000 UTC m=+1383.995601117"
Sep 08 15:35:43.138151 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:43.138117    2568 generic.go:356] "Generic (PLEG): container finished" podID="0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" containerID="2b827e4414962680e7b54403d64c9b360d32edae79f9f88c100983d78ac7445e" exitCode=0
Sep 08 15:35:43.138549 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:43.138202    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7769-6b9828" event={"ID":"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb","Type":"ContainerDied","Data":"2b827e4414962680e7b54403d64c9b360d32edae79f9f88c100983d78ac7445e"}
Sep 08 15:35:44.271544 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.271519    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:44.292056 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.292027    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s5bh\" (UniqueName: \"kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh\") pod \"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb\" (UID: \"0a681e81-9b10-456d-ba5f-bb4b6e9a66bb\") "
Sep 08 15:35:44.294052 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.294027    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh" (OuterVolumeSpecName: "kube-api-access-8s5bh") pod "0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" (UID: "0a681e81-9b10-456d-ba5f-bb4b6e9a66bb"). InnerVolumeSpecName "kube-api-access-8s5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:44.305007 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.304990    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7769-6b9828_0a681e81-9b10-456d-ba5f-bb4b6e9a66bb/mt-curl-7769-6b9828/0.log"
Sep 08 15:35:44.327901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.327877    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7769-6b9828"]
Sep 08 15:35:44.331234 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.331215    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7769-6b9828"]
Sep 08 15:35:44.392889 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.392865    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5bh\" (UniqueName: \"kubernetes.io/projected/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb-kube-api-access-8s5bh\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:35:44.446168 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.446107    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7769-aa2a5d"]
Sep 08 15:35:44.446539 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.446524    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" containerName="mt-curl-7769-6b9828"
Sep 08 15:35:44.446590 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.446543    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" containerName="mt-curl-7769-6b9828"
Sep 08 15:35:44.446635 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.446625    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" containerName="mt-curl-7769-6b9828"
Sep 08 15:35:44.449620 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.449606    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:44.455626 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.455590    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7769-aa2a5d"]
Sep 08 15:35:44.493584 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.493555    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fff\" (UniqueName: \"kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff\") pod \"mt-curl-7769-aa2a5d\" (UID: \"b64ee90a-a637-47f2-aa20-28f1ac87fd32\") " pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:44.594236 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.594193    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fff\" (UniqueName: \"kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff\") pod \"mt-curl-7769-aa2a5d\" (UID: \"b64ee90a-a637-47f2-aa20-28f1ac87fd32\") " pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:44.601805 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.601780    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fff\" (UniqueName: \"kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff\") pod \"mt-curl-7769-aa2a5d\" (UID: \"b64ee90a-a637-47f2-aa20-28f1ac87fd32\") " pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:44.675882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.675851    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a681e81-9b10-456d-ba5f-bb4b6e9a66bb" path="/var/lib/kubelet/pods/0a681e81-9b10-456d-ba5f-bb4b6e9a66bb/volumes"
Sep 08 15:35:44.760060 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:44.759982    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:45.031547 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:45.031504    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7769-aa2a5d"]
Sep 08 15:35:45.150922 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:45.150894    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-6b9828"
Sep 08 15:35:45.151030 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:45.150927    2568 scope.go:117] "RemoveContainer" containerID="2b827e4414962680e7b54403d64c9b360d32edae79f9f88c100983d78ac7445e"
Sep 08 15:35:45.152365 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:45.152337    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7769-aa2a5d" event={"ID":"b64ee90a-a637-47f2-aa20-28f1ac87fd32","Type":"ContainerStarted","Data":"b8c97abd3f13f6c6da01a75638eb56d2fe74d29352a42a1529b8e12382cd6e0a"}
Sep 08 15:35:46.160316 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:46.160265    2568 generic.go:356] "Generic (PLEG): container finished" podID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" containerID="5468a6ea77798c30646b98e98f57fa572095bb9718bda4875f8b70fbce54db9c" exitCode=0
Sep 08 15:35:46.160741 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:46.160336    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7769-aa2a5d" event={"ID":"b64ee90a-a637-47f2-aa20-28f1ac87fd32","Type":"ContainerDied","Data":"5468a6ea77798c30646b98e98f57fa572095bb9718bda4875f8b70fbce54db9c"}
Sep 08 15:35:46.699105 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:46.699067    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:35:47.306075 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.306054    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:47.319266 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.319246    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26fff\" (UniqueName: \"kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff\") pod \"b64ee90a-a637-47f2-aa20-28f1ac87fd32\" (UID: \"b64ee90a-a637-47f2-aa20-28f1ac87fd32\") "
Sep 08 15:35:47.321210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.321178    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff" (OuterVolumeSpecName: "kube-api-access-26fff") pod "b64ee90a-a637-47f2-aa20-28f1ac87fd32" (UID: "b64ee90a-a637-47f2-aa20-28f1ac87fd32"). InnerVolumeSpecName "kube-api-access-26fff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:35:47.346883 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.346859    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7769-aa2a5d_b64ee90a-a637-47f2-aa20-28f1ac87fd32/mt-curl-7769-aa2a5d/0.log"
Sep 08 15:35:47.415550 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.415518    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7769-aa2a5d"]
Sep 08 15:35:47.420124 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.420070    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fff\" (UniqueName: \"kubernetes.io/projected/b64ee90a-a637-47f2-aa20-28f1ac87fd32-kube-api-access-26fff\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:35:47.429167 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:47.429143    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7769-aa2a5d"]
Sep 08 15:35:48.144028 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:48.143993    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:35:48.175484 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:48.175456    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:48.175656 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:48.175468    2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c97abd3f13f6c6da01a75638eb56d2fe74d29352a42a1529b8e12382cd6e0a"
Sep 08 15:35:48.186120 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:35:48.186087    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7769-aa2a5d\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:48.674976 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:35:48.674939    2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7769-aa2a5d\" is forbidden: User \"system:node:ip-10-0-134-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-180.ec2.internal' and this object" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" pod="opendatahub/mt-curl-7769-aa2a5d"
Sep 08 15:35:48.676393 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:48.676370    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" path="/var/lib/kubelet/pods/b64ee90a-a637-47f2-aa20-28f1ac87fd32/volumes"
Sep 08 15:35:49.507766 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:49.507733    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:35:50.419025 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:35:50.418992    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:36:03.817474 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.817439    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:36:03.817893 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.817825    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" containerName="mt-curl-7769-aa2a5d"
Sep 08 15:36:03.817893 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.817836    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" containerName="mt-curl-7769-aa2a5d"
Sep 08 15:36:03.817984 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.817900    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b64ee90a-a637-47f2-aa20-28f1ac87fd32" containerName="mt-curl-7769-aa2a5d"
Sep 08 15:36:03.821220 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.821203    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.823532 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.823508    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fc867c\"/\"rate-test-model-fc867c-kserve-self-signed-certs\""
Sep 08 15:36:03.829728 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.829708    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:36:03.858589 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858564    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.858742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858638    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.858742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858662    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.858742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858682    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.858742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858707    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.858742 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.858729    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cckt\" (UniqueName: \"kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.959750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959712    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.959943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959759    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.959943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959790    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cckt\" (UniqueName: \"kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.959943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959819    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.959943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959924    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.960178 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.959959    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.960272 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.960247    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.960376 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.960321    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.960376 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.960333    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.962132 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.962114    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.962302 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.962275    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:03.968558 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:03.968529    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cckt\" (UniqueName: \"kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt\") pod \"rate-test-model-fc867c-kserve-85bb7846f5-72t9k\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") " pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:04.131439 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.131410    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:04.403586 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.403555    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:36:04.405022 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:36:04.404997    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f2d73e7_f270_43c1_9c63_1fdb5f3fb5db.slice/crio-925e1dd2db3905c3e3cdf654f331ab60f4903e69e1b0255b7aeadffe00407589 WatchSource:0}: Error finding container 925e1dd2db3905c3e3cdf654f331ab60f4903e69e1b0255b7aeadffe00407589: Status 404 returned error can't find the container with id 925e1dd2db3905c3e3cdf654f331ab60f4903e69e1b0255b7aeadffe00407589
Sep 08 15:36:04.527568 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.527511    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" containerName="main" containerID="cri-o://882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256" gracePeriod=45
Sep 08 15:36:04.633211 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.633168    2568 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 15:36:04.766763 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.766737    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:36:04.866850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.866817    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.866862    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzf57\" (UniqueName: \"kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.866890    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.866958    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.866978    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867004    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm\") pod \"d966e063-e13e-4f82-b001-d5546af3daaa\" (UID: \"d966e063-e13e-4f82-b001-d5546af3daaa\") "
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867120    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home" (OuterVolumeSpecName: "home") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867152    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:04.867249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867240    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:04.867611 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867319    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:04.867611 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.867339    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:04.868945 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.868920    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm" (OuterVolumeSpecName: "dshm") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:04.869014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.868945    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:04.869014 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.868996    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57" (OuterVolumeSpecName: "kube-api-access-wzf57") pod "d966e063-e13e-4f82-b001-d5546af3daaa" (UID: "d966e063-e13e-4f82-b001-d5546af3daaa"). InnerVolumeSpecName "kube-api-access-wzf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:36:04.968760 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.968674    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d966e063-e13e-4f82-b001-d5546af3daaa-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:04.968760 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.968702    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:04.968760 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.968710    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d966e063-e13e-4f82-b001-d5546af3daaa-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:04.968760 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:04.968719    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzf57\" (UniqueName: \"kubernetes.io/projected/d966e063-e13e-4f82-b001-d5546af3daaa-kube-api-access-wzf57\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.274664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.274567    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" event={"ID":"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db","Type":"ContainerStarted","Data":"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"}
Sep 08 15:36:05.274664 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.274610    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" event={"ID":"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db","Type":"ContainerStarted","Data":"925e1dd2db3905c3e3cdf654f331ab60f4903e69e1b0255b7aeadffe00407589"}
Sep 08 15:36:05.275974 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.275948    2568 generic.go:356] "Generic (PLEG): container finished" podID="d966e063-e13e-4f82-b001-d5546af3daaa" containerID="882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256" exitCode=0
Sep 08 15:36:05.276102 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.275985    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"
Sep 08 15:36:05.276102 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.276019    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" event={"ID":"d966e063-e13e-4f82-b001-d5546af3daaa","Type":"ContainerDied","Data":"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"}
Sep 08 15:36:05.276102 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.276045    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z" event={"ID":"d966e063-e13e-4f82-b001-d5546af3daaa","Type":"ContainerDied","Data":"1773bcea9ce7e363865474c33e0088e5325c2f766b7f22c76702f5d437f3d449"}
Sep 08 15:36:05.276102 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.276061    2568 scope.go:117] "RemoveContainer" containerID="882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"
Sep 08 15:36:05.286108 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.286091    2568 scope.go:117] "RemoveContainer" containerID="882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"
Sep 08 15:36:05.286374 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:36:05.286356    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256\": container with ID starting with 882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256 not found: ID does not exist" containerID="882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"
Sep 08 15:36:05.286426 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.286380    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256"} err="failed to get container status \"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256\": rpc error: code = NotFound desc = could not find container \"882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256\": container with ID starting with 882295769f0e23ed2119a69bb3182e08b694fe734d81f139acd87b0b2638b256 not found: ID does not exist"
Sep 08 15:36:05.296830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.296784    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" podStartSLOduration=2.296770586 podStartE2EDuration="2.296770586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:05.295090795 +0000 UTC m=+1407.137888017" watchObservedRunningTime="2026-09-08 15:36:05.296770586 +0000 UTC m=+1407.139567843"
Sep 08 15:36:05.309885 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.309850    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:36:05.312109 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.312091    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/sub-test-model-fc867c-kserve-568695bbcc-6q98z"]
Sep 08 15:36:05.438608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.438573    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" podUID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" containerName="main" containerID="cri-o://d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452" gracePeriod=45
Sep 08 15:36:05.697051 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.697028    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:36:05.776201 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776168    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776206    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776225    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776260    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ggwz\" (UniqueName: \"kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776287    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776347    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm\") pod \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\" (UID: \"41f5a744-0f09-4e73-91a7-a4d4e74d3420\") "
Sep 08 15:36:05.776716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776475    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:05.776716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776554    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home" (OuterVolumeSpecName: "home") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:05.776716 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776615    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache" (OuterVolumeSpecName: "model-cache") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:05.776870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776771    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.776870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776790    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.776870 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.776805    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.778411 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.778387    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:05.778514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.778448    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz" (OuterVolumeSpecName: "kube-api-access-4ggwz") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "kube-api-access-4ggwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:36:05.778583 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.778570    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm" (OuterVolumeSpecName: "dshm") pod "41f5a744-0f09-4e73-91a7-a4d4e74d3420" (UID: "41f5a744-0f09-4e73-91a7-a4d4e74d3420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:36:05.877731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.877709    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggwz\" (UniqueName: \"kubernetes.io/projected/41f5a744-0f09-4e73-91a7-a4d4e74d3420-kube-api-access-4ggwz\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.877731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.877731    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41f5a744-0f09-4e73-91a7-a4d4e74d3420-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:05.878080 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:05.877740    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41f5a744-0f09-4e73-91a7-a4d4e74d3420-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:06.284446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.284365    2568 generic.go:356] "Generic (PLEG): container finished" podID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" containerID="d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452" exitCode=0
Sep 08 15:36:06.284446 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.284409    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"
Sep 08 15:36:06.284660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.284451    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" event={"ID":"41f5a744-0f09-4e73-91a7-a4d4e74d3420","Type":"ContainerDied","Data":"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"}
Sep 08 15:36:06.284660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.284492    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c" event={"ID":"41f5a744-0f09-4e73-91a7-a4d4e74d3420","Type":"ContainerDied","Data":"695f4b5a37473130747c6acc3aeb6b8729fe1ffb4172284b18fe9c891238f293"}
Sep 08 15:36:06.284660 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.284514    2568 scope.go:117] "RemoveContainer" containerID="d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"
Sep 08 15:36:06.295906 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.295884    2568 scope.go:117] "RemoveContainer" containerID="d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"
Sep 08 15:36:06.296255 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:36:06.296230    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452\": container with ID starting with d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452 not found: ID does not exist" containerID="d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"
Sep 08 15:36:06.296349 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.296275    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452"} err="failed to get container status \"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452\": rpc error: code = NotFound desc = could not find container \"d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452\": container with ID starting with d978395bad9842e37f1693de844f66624d4032784a4c618569ccc2e0d30d6452 not found: ID does not exist"
Sep 08 15:36:06.308019 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.307996    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:36:06.312106 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.312083    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/sub-test-model-565cd7-kserve-56bbfdf979-bkt6c"]
Sep 08 15:36:06.675895 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.675856    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" path="/var/lib/kubelet/pods/41f5a744-0f09-4e73-91a7-a4d4e74d3420/volumes"
Sep 08 15:36:06.676273 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:06.676259    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" path="/var/lib/kubelet/pods/d966e063-e13e-4f82-b001-d5546af3daaa/volumes"
Sep 08 15:36:07.823070 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:07.823036    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:36:14.131841 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:14.131808    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:14.132464 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:14.131886    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:14.144154 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:14.144132    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:14.340733 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:14.340699    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:36:29.902599 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:29.902558    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:36:45.916381 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:45.916343    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:36:45.916913 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:45.916559    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" podUID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" containerName="maas-api" containerID="cri-o://df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881" gracePeriod=30
Sep 08 15:36:46.169656 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.169597    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:36:46.212288 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.212263    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls\") pod \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") "
Sep 08 15:36:46.212444 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.212363    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzj8z\" (UniqueName: \"kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z\") pod \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") "
Sep 08 15:36:46.212444 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.212393    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs\") pod \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") "
Sep 08 15:36:46.212444 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.212436    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca\") pod \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\" (UID: \"4f750439-2fe2-42cc-9e3a-4dfe9942e122\") "
Sep 08 15:36:46.212800 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.212772    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f750439-2fe2-42cc-9e3a-4dfe9942e122" (UID: "4f750439-2fe2-42cc-9e3a-4dfe9942e122"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:36:46.214510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.214475    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f750439-2fe2-42cc-9e3a-4dfe9942e122" (UID: "4f750439-2fe2-42cc-9e3a-4dfe9942e122"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:46.214608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.214525    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z" (OuterVolumeSpecName: "kube-api-access-lzj8z") pod "4f750439-2fe2-42cc-9e3a-4dfe9942e122" (UID: "4f750439-2fe2-42cc-9e3a-4dfe9942e122"). InnerVolumeSpecName "kube-api-access-lzj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:36:46.214608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.214594    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4f750439-2fe2-42cc-9e3a-4dfe9942e122" (UID: "4f750439-2fe2-42cc-9e3a-4dfe9942e122"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:36:46.313150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.313121    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f750439-2fe2-42cc-9e3a-4dfe9942e122-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:46.313150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.313144    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:46.313150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.313154    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzj8z\" (UniqueName: \"kubernetes.io/projected/4f750439-2fe2-42cc-9e3a-4dfe9942e122-kube-api-access-lzj8z\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:46.313385 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.313161    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f750439-2fe2-42cc-9e3a-4dfe9942e122-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:36:46.510315 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.510214    2568 generic.go:356] "Generic (PLEG): container finished" podID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" containerID="df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881" exitCode=0
Sep 08 15:36:46.510315 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.510258    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"
Sep 08 15:36:46.510536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.510315    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" event={"ID":"4f750439-2fe2-42cc-9e3a-4dfe9942e122","Type":"ContainerDied","Data":"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"}
Sep 08 15:36:46.510536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.510372    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv" event={"ID":"4f750439-2fe2-42cc-9e3a-4dfe9942e122","Type":"ContainerDied","Data":"6c3b1c3d68c00eabd9a4c91a79db658b02b2d8a665122d522111e90185ded11d"}
Sep 08 15:36:46.510536 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.510403    2568 scope.go:117] "RemoveContainer" containerID="df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"
Sep 08 15:36:46.520250 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.520232    2568 scope.go:117] "RemoveContainer" containerID="df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"
Sep 08 15:36:46.520538 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:36:46.520519    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881\": container with ID starting with df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881 not found: ID does not exist" containerID="df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"
Sep 08 15:36:46.520616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.520548    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881"} err="failed to get container status \"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881\": rpc error: code = NotFound desc = could not find container \"df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881\": container with ID starting with df111e686f35b078b503ce305952cd05d6d7c739f4dfa6ac02a35be4504e6881 not found: ID does not exist"
Sep 08 15:36:46.532999 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.532978    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:36:46.537123 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.537103    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e34cd-75fd9df88f-gwllv"]
Sep 08 15:36:46.675458 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:46.675428    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" path="/var/lib/kubelet/pods/4f750439-2fe2-42cc-9e3a-4dfe9942e122/volumes"
Sep 08 15:36:50.101830 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:50.101794    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:36:56.626607 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.626568    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:36:56.627075 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627063    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" containerName="main"
Sep 08 15:36:56.627128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627082    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" containerName="main"
Sep 08 15:36:56.627128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627107    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" containerName="main"
Sep 08 15:36:56.627128 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627116    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" containerName="main"
Sep 08 15:36:56.627237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627140    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" containerName="maas-api"
Sep 08 15:36:56.627237 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627149    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" containerName="maas-api"
Sep 08 15:36:56.627332 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627244    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d966e063-e13e-4f82-b001-d5546af3daaa" containerName="main"
Sep 08 15:36:56.627332 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627260    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f750439-2fe2-42cc-9e3a-4dfe9942e122" containerName="maas-api"
Sep 08 15:36:56.627332 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.627270    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="41f5a744-0f09-4e73-91a7-a4d4e74d3420" containerName="main"
Sep 08 15:36:56.631842 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.631823    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.634201 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.634178    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"openshift-service-ca.crt\""
Sep 08 15:36:56.634330 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.634228    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"default-dockercfg-gllms\""
Sep 08 15:36:56.635048 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.635030    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"rate-test-model-565cd7-kserve-self-signed-certs\""
Sep 08 15:36:56.635149 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.635130    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-565cd7\"/\"kube-root-ca.crt\""
Sep 08 15:36:56.640934 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.640911    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:36:56.704784 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704751    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.704784 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704784    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.704996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704811    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.704996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704893    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmtpd\" (UniqueName: \"kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.704996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704955    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.704996 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.704996    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806484    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806530    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806556    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806589    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmtpd\" (UniqueName: \"kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806636    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.806707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.806662    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.807029 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.807007    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.807103 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.807040    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.807144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.807094    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.808758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.808739    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.809025 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.809007    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.814903 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.814874    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmtpd\" (UniqueName: \"kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd\") pod \"rate-test-model-565cd7-kserve-65b6cf7f47-v62g7\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") " pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:56.941350 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:56.941244    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:36:57.224407 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:57.224380    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:36:57.575195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:57.575104    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" event={"ID":"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e","Type":"ContainerStarted","Data":"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"}
Sep 08 15:36:57.575195 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:57.575144    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" event={"ID":"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e","Type":"ContainerStarted","Data":"e01e799396de640815ca32429b03213d2d7c5d991a70189ca370008b707c3d1a"}
Sep 08 15:36:57.592573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:36:57.592524    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" podStartSLOduration=1.59251008 podStartE2EDuration="1.59251008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:36:57.592028675 +0000 UTC m=+1459.434825897" watchObservedRunningTime="2026-09-08 15:36:57.59251008 +0000 UTC m=+1459.435307294"
Sep 08 15:37:03.402994 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.402962    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:37:03.403535 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.403148    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" podUID="67fc746a-1eed-4958-b04d-2e826855e606" containerName="payload-processing" containerID="cri-o://25d3c423deb06bb022f072f206af3cda4e51a51b532ff2e0416fa9e4646efb95" gracePeriod=30
Sep 08 15:37:03.610581 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.610549    2568 generic.go:356] "Generic (PLEG): container finished" podID="67fc746a-1eed-4958-b04d-2e826855e606" containerID="25d3c423deb06bb022f072f206af3cda4e51a51b532ff2e0416fa9e4646efb95" exitCode=0
Sep 08 15:37:03.610754 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.610624    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" event={"ID":"67fc746a-1eed-4958-b04d-2e826855e606","Type":"ContainerDied","Data":"25d3c423deb06bb022f072f206af3cda4e51a51b532ff2e0416fa9e4646efb95"}
Sep 08 15:37:03.650717 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.650695    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:37:03.767147 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.767034    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume\") pod \"67fc746a-1eed-4958-b04d-2e826855e606\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") "
Sep 08 15:37:03.767147 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.767126    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txc4l\" (UniqueName: \"kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l\") pod \"67fc746a-1eed-4958-b04d-2e826855e606\" (UID: \"67fc746a-1eed-4958-b04d-2e826855e606\") "
Sep 08 15:37:03.767487 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.767462    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67fc746a-1eed-4958-b04d-2e826855e606" (UID: "67fc746a-1eed-4958-b04d-2e826855e606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:37:03.769401 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.769372    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l" (OuterVolumeSpecName: "kube-api-access-txc4l") pod "67fc746a-1eed-4958-b04d-2e826855e606" (UID: "67fc746a-1eed-4958-b04d-2e826855e606"). InnerVolumeSpecName "kube-api-access-txc4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:37:03.868532 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.868499    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fc746a-1eed-4958-b04d-2e826855e606-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:03.868532 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:03.868523    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txc4l\" (UniqueName: \"kubernetes.io/projected/67fc746a-1eed-4958-b04d-2e826855e606-kube-api-access-txc4l\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:04.618475 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.618443    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"
Sep 08 15:37:04.618475 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.618462    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk" event={"ID":"67fc746a-1eed-4958-b04d-2e826855e606","Type":"ContainerDied","Data":"322f3626da7d90d32df7b974273767ac85a7f53a7383ee8063f6140a2137f13f"}
Sep 08 15:37:04.618949 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.618561    2568 scope.go:117] "RemoveContainer" containerID="25d3c423deb06bb022f072f206af3cda4e51a51b532ff2e0416fa9e4646efb95"
Sep 08 15:37:04.640750 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.640720    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:37:04.646116 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.646084    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-020a18-77db95b885-9cfvk"]
Sep 08 15:37:04.676919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:04.676893    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67fc746a-1eed-4958-b04d-2e826855e606" path="/var/lib/kubelet/pods/67fc746a-1eed-4958-b04d-2e826855e606/volumes"
Sep 08 15:37:06.942357 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:06.942321    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:06.942816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:06.942370    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:06.954964 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:06.954933    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:07.649981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:07.649955    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:09.597226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:09.597185    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:37:20.101882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:20.101846    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:37:21.082044 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.082009    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:37:21.082272 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.082246    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" podUID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" containerName="payload-processing" containerID="cri-o://a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c" gracePeriod=30
Sep 08 15:37:21.337768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.337709    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:37:21.428357 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.428325    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume\") pod \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") "
Sep 08 15:37:21.428521 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.428469    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78f4t\" (UniqueName: \"kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t\") pod \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\" (UID: \"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3\") "
Sep 08 15:37:21.428705 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.428683    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" (UID: "df0bb4ec-7bfe-48e8-bb4a-29049c343fa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:37:21.428778 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.428762    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:21.430617 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.430590    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t" (OuterVolumeSpecName: "kube-api-access-78f4t") pod "df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" (UID: "df0bb4ec-7bfe-48e8-bb4a-29049c343fa3"). InnerVolumeSpecName "kube-api-access-78f4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:37:21.529479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.529440    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78f4t\" (UniqueName: \"kubernetes.io/projected/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3-kube-api-access-78f4t\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:21.714658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.714623    2568 generic.go:356] "Generic (PLEG): container finished" podID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" containerID="a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c" exitCode=0
Sep 08 15:37:21.714844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.714668    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"
Sep 08 15:37:21.714844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.714700    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" event={"ID":"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3","Type":"ContainerDied","Data":"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"}
Sep 08 15:37:21.714844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.714752    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg" event={"ID":"df0bb4ec-7bfe-48e8-bb4a-29049c343fa3","Type":"ContainerDied","Data":"3b223c3b812db8f7d8274949f947a19f5275265b7388fe9a1117d7770309fd14"}
Sep 08 15:37:21.714844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.714776    2568 scope.go:117] "RemoveContainer" containerID="a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"
Sep 08 15:37:21.726496 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.726472    2568 scope.go:117] "RemoveContainer" containerID="a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"
Sep 08 15:37:21.726845 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:37:21.726810    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c\": container with ID starting with a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c not found: ID does not exist" containerID="a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"
Sep 08 15:37:21.726929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.726849    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c"} err="failed to get container status \"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c\": rpc error: code = NotFound desc = could not find container \"a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c\": container with ID starting with a768629754eb0e56774cb215f64f677d8890ce3b09594529c2d815f4cbdddf3c not found: ID does not exist"
Sep 08 15:37:21.739451 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.739425    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:37:21.743319 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:21.743281    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d62b4e-58456dd66f-98slg"]
Sep 08 15:37:22.677150 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:22.677112    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" path="/var/lib/kubelet/pods/df0bb4ec-7bfe-48e8-bb4a-29049c343fa3/volumes"
Sep 08 15:37:30.812410 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:30.812375    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:37:36.435395 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:36.435358    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:37:37.446704 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:37.446673    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:37:38.727193 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:38.727167    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:37:38.733657 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:38.733634    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:37:39.106768 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:39.106693    2568 scope.go:117] "RemoveContainer" containerID="b3558b6778f584333b49da0bad95f6a4e1ac23052040879da0dfcc8171aefd5d"
Sep 08 15:37:39.115329 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:39.115275    2568 scope.go:117] "RemoveContainer" containerID="6577c332d42f0e3dd3a7d0e62f187ff1da7aefaa00edcf6be7c5f731f5a1902e"
Sep 08 15:37:39.123480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:39.123459    2568 scope.go:117] "RemoveContainer" containerID="a420e18bd1d660be4f96b77cad11295c17d992a396bb6c036d3aa70f33409afe"
Sep 08 15:37:39.131388 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:39.131277    2568 scope.go:117] "RemoveContainer" containerID="e0c32ec9590a2b877737aa5e5f98a2a2632954eb2347259b492b8fa1f7a785a9"
Sep 08 15:37:46.509993 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:46.509953    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:37:51.456684 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.456625    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" podUID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" containerName="main" containerID="cri-o://bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea" gracePeriod=45
Sep 08 15:37:51.707928 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.707861    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:37:51.793527 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793498    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793559    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793579    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cckt\" (UniqueName: \"kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793632    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793653    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793691 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793677    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home\") pod \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\" (UID: \"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db\") "
Sep 08 15:37:51.793961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793808    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:51.793961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793864    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:51.794062 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.793978    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home" (OuterVolumeSpecName: "home") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:51.794062 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.794022    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.794062 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.794040    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.795900 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.795869    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:37:51.795998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.795905    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm" (OuterVolumeSpecName: "dshm") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:51.795998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.795919    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt" (OuterVolumeSpecName: "kube-api-access-8cckt") pod "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" (UID: "9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db"). InnerVolumeSpecName "kube-api-access-8cckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:37:51.884479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.884447    2568 generic.go:356] "Generic (PLEG): container finished" podID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" containerID="bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea" exitCode=0
Sep 08 15:37:51.884620 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.884482    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" event={"ID":"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db","Type":"ContainerDied","Data":"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"}
Sep 08 15:37:51.884620 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.884494    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"
Sep 08 15:37:51.884620 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.884509    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k" event={"ID":"9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db","Type":"ContainerDied","Data":"925e1dd2db3905c3e3cdf654f331ab60f4903e69e1b0255b7aeadffe00407589"}
Sep 08 15:37:51.884620 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.884530    2568 scope.go:117] "RemoveContainer" containerID="bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"
Sep 08 15:37:51.893983 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.893966    2568 scope.go:117] "RemoveContainer" containerID="bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"
Sep 08 15:37:51.894270 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:37:51.894250    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea\": container with ID starting with bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea not found: ID does not exist" containerID="bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"
Sep 08 15:37:51.894367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.894281    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea"} err="failed to get container status \"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea\": rpc error: code = NotFound desc = could not find container \"bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea\": container with ID starting with bbdfce6097d8d21a06611b1a3477f23f43866cedfab97cd4392547e290236eea not found: ID does not exist"
Sep 08 15:37:51.894465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.894449    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cckt\" (UniqueName: \"kubernetes.io/projected/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-kube-api-access-8cckt\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.894507 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.894468    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.894507 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.894478    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.894507 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.894486    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:51.910541 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.910516    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:37:51.913212 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:51.913193    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fc867c/rate-test-model-fc867c-kserve-85bb7846f5-72t9k"]
Sep 08 15:37:52.466171 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.466130    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" podUID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" containerName="main" containerID="cri-o://fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9" gracePeriod=45
Sep 08 15:37:52.676518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.676489    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" path="/var/lib/kubelet/pods/9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db/volumes"
Sep 08 15:37:52.721498 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.721442    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:52.802178 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802152    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802220    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802249    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802265    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802338 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802331    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802358    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmtpd\" (UniqueName: \"kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd\") pod \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\" (UID: \"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e\") "
Sep 08 15:37:52.802602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802539    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home" (OuterVolumeSpecName: "home") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:52.802602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802551    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:52.802602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802560    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:52.802745 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802721    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.802782 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802745    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.802782 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.802763    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.804383 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.804356    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm" (OuterVolumeSpecName: "dshm") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:37:52.804473 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.804420    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:37:52.804473 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.804430    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd" (OuterVolumeSpecName: "kube-api-access-gmtpd") pod "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" (UID: "fe75e30b-ca16-4ff8-9455-54fc3ba44a1e"). InnerVolumeSpecName "kube-api-access-gmtpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:37:52.891309 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.891265    2568 generic.go:356] "Generic (PLEG): container finished" podID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" containerID="fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9" exitCode=0
Sep 08 15:37:52.891472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.891333    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"
Sep 08 15:37:52.891472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.891357    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" event={"ID":"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e","Type":"ContainerDied","Data":"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"}
Sep 08 15:37:52.891472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.891404    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7" event={"ID":"fe75e30b-ca16-4ff8-9455-54fc3ba44a1e","Type":"ContainerDied","Data":"e01e799396de640815ca32429b03213d2d7c5d991a70189ca370008b707c3d1a"}
Sep 08 15:37:52.891472 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.891433    2568 scope.go:117] "RemoveContainer" containerID="fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"
Sep 08 15:37:52.901331 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.901286    2568 scope.go:117] "RemoveContainer" containerID="fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"
Sep 08 15:37:52.901611 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:37:52.901592    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9\": container with ID starting with fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9 not found: ID does not exist" containerID="fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"
Sep 08 15:37:52.901683 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.901625    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9"} err="failed to get container status \"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9\": rpc error: code = NotFound desc = could not find container \"fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9\": container with ID starting with fa825b5f354100387256de82185c1b8b8dc0b4d355a8ad2e48eedaf7506ef0e9 not found: ID does not exist"
Sep 08 15:37:52.904013 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.903992    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.904074 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.904018    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmtpd\" (UniqueName: \"kubernetes.io/projected/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-kube-api-access-gmtpd\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.904074 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.904032    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:37:52.918478 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.918452    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:37:52.922497 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:52.922477    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-565cd7/rate-test-model-565cd7-kserve-65b6cf7f47-v62g7"]
Sep 08 15:37:54.681585 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:37:54.681553    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" path="/var/lib/kubelet/pods/fe75e30b-ca16-4ff8-9455-54fc3ba44a1e/volumes"
Sep 08 15:38:11.242639 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.242599    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243135    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243150    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243168    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" containerName="payload-processing"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243177    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" containerName="payload-processing"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243188    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243197    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243218    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fc746a-1eed-4958-b04d-2e826855e606" containerName="payload-processing"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243227    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fc746a-1eed-4958-b04d-2e826855e606" containerName="payload-processing"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243332    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0bb4ec-7bfe-48e8-bb4a-29049c343fa3" containerName="payload-processing"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243347    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2d73e7-f270-43c1-9c63-1fdb5f3fb5db" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243357    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe75e30b-ca16-4ff8-9455-54fc3ba44a1e" containerName="main"
Sep 08 15:38:11.245008 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.243366    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fc746a-1eed-4958-b04d-2e826855e606" containerName="payload-processing"
Sep 08 15:38:11.246708 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.246683    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.248744 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.248717    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-30cc8d-serving-cert\""
Sep 08 15:38:11.263477 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.263448    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:38:11.363805 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.363771    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.363805 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.363806    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wt4k\" (UniqueName: \"kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.364034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.363895    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.364034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.363945    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.465144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.465106    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.465144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.465150    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.465414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.465218    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.465414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.465247    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wt4k\" (UniqueName: \"kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.465989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.465961    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.467723 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.467694    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.467834 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.467767    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.474625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.474600    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wt4k\" (UniqueName: \"kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k\") pod \"maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.559653 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.559561    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:11.846219 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.846185    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:38:11.846619 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:38:11.846585    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6064711_6dc9_4d6e_9642_a789f1cde8ec.slice/crio-fe65fdd4e8025b1a73f1861ebea725c9195bc2914e4d1fb5d5656f40939135ff WatchSource:0}: Error finding container fe65fdd4e8025b1a73f1861ebea725c9195bc2914e4d1fb5d5656f40939135ff: Status 404 returned error can't find the container with id fe65fdd4e8025b1a73f1861ebea725c9195bc2914e4d1fb5d5656f40939135ff
Sep 08 15:38:11.996971 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:11.996935    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" event={"ID":"c6064711-6dc9-4d6e-9642-a789f1cde8ec","Type":"ContainerStarted","Data":"fe65fdd4e8025b1a73f1861ebea725c9195bc2914e4d1fb5d5656f40939135ff"}
Sep 08 15:38:12.056843 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.056814    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:38:12.061244 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.061213    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.063467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.063446    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-30cc8d\""
Sep 08 15:38:12.063559 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.063489    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-30cc8d-dockercfg-pnwk9\""
Sep 08 15:38:12.072386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.072363    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:38:12.171302 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.171265    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.171425 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.171398    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgr68\" (UniqueName: \"kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.272951 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.272917    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.273353 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.272974    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgr68\" (UniqueName: \"kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.273666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.273644    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.281943 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.281923    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgr68\" (UniqueName: \"kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68\") pod \"payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.373467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.373432    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:12.643343 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:12.643317    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:38:12.645125 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:38:12.645095    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e63d677_b2fb_4b10_8214_d55daba0e041.slice/crio-f053084c1dce077cd63a4da643dab8ce80a69424880b1c91a11277ea9d1a3663 WatchSource:0}: Error finding container f053084c1dce077cd63a4da643dab8ce80a69424880b1c91a11277ea9d1a3663: Status 404 returned error can't find the container with id f053084c1dce077cd63a4da643dab8ce80a69424880b1c91a11277ea9d1a3663
Sep 08 15:38:13.004682 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.004580    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" event={"ID":"c6064711-6dc9-4d6e-9642-a789f1cde8ec","Type":"ContainerStarted","Data":"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"}
Sep 08 15:38:13.004682 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.004646    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:13.006230 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.006199    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" event={"ID":"5e63d677-b2fb-4b10-8214-d55daba0e041","Type":"ContainerStarted","Data":"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"}
Sep 08 15:38:13.006396 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.006248    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" event={"ID":"5e63d677-b2fb-4b10-8214-d55daba0e041","Type":"ContainerStarted","Data":"f053084c1dce077cd63a4da643dab8ce80a69424880b1c91a11277ea9d1a3663"}
Sep 08 15:38:13.006396 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.006269    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:13.025413 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.025368    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" podStartSLOduration=1.732340621 podStartE2EDuration="2.025350201s" totalImagesPullingTime="293.00958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:13.024186131 +0000 UTC m=+1534.866983367" watchObservedRunningTime="2026-09-08 15:38:13.025350201 +0000 UTC m=+1534.868147426"
Sep 08 15:38:13.045961 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:13.045909    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" podStartSLOduration=1.045895232 podStartE2EDuration="1.045895232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:13.042978635 +0000 UTC m=+1534.885775860" watchObservedRunningTime="2026-09-08 15:38:13.045895232 +0000 UTC m=+1534.888692507"
Sep 08 15:38:17.703727 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:17.703689    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:38:19.017373 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:19.017341    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:38:24.014079 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:24.014048    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:38:40.303010 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:40.302971    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4_5e63d677-b2fb-4b10-8214-d55daba0e041/payload-processing/0.log"
Sep 08 15:38:47.228663 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.228624    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:38:47.232231 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.232212    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.236668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.236641    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-014f36\"/\"kube-root-ca.crt\""
Sep 08 15:38:47.236901 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.236878    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-014f36\"/\"ipp-route-014f36-kserve-self-signed-certs\""
Sep 08 15:38:47.237109 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.237085    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-014f36\"/\"default-dockercfg-4p5tw\""
Sep 08 15:38:47.237442 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.237416    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-014f36\"/\"openshift-service-ca.crt\""
Sep 08 15:38:47.247689 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.247665    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:38:47.373604 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373568    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.373822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373637    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.373822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373704    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.373822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373784    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.373822 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373820    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjzf\" (UniqueName: \"kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.373989 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.373845    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475026 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.474980    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjzf\" (UniqueName: \"kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475026 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475026    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475288 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475158    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475288 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475255    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475319    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475436 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475397    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475523 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475394    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475523 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475496    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.475668 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.475643    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.477583 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.477559    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.477715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.477698    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.485169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.485112    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjzf\" (UniqueName: \"kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf\") pod \"ipp-route-014f36-kserve-79fd5fd8cf-5dpgn\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") " pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.543367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.543327    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:47.817778 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:47.817752    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:38:47.819881 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:38:47.819851    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04592b51_dfee_49f1_bb96_c0b62717aa7c.slice/crio-8b84543a1c011da6c4ba6d228e9a78f96fc5029459a0c03b59e75161d16b37e7 WatchSource:0}: Error finding container 8b84543a1c011da6c4ba6d228e9a78f96fc5029459a0c03b59e75161d16b37e7: Status 404 returned error can't find the container with id 8b84543a1c011da6c4ba6d228e9a78f96fc5029459a0c03b59e75161d16b37e7
Sep 08 15:38:48.205539 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:48.205502    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" event={"ID":"04592b51-dfee-49f1-bb96-c0b62717aa7c","Type":"ContainerStarted","Data":"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"}
Sep 08 15:38:48.205715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:48.205545    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" event={"ID":"04592b51-dfee-49f1-bb96-c0b62717aa7c","Type":"ContainerStarted","Data":"8b84543a1c011da6c4ba6d228e9a78f96fc5029459a0c03b59e75161d16b37e7"}
Sep 08 15:38:48.227712 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:48.227651    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" podStartSLOduration=1.227629466 podStartE2EDuration="1.227629466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:38:48.226243105 +0000 UTC m=+1570.069040327" watchObservedRunningTime="2026-09-08 15:38:48.227629466 +0000 UTC m=+1570.070426690"
Sep 08 15:38:52.902022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:52.901989    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:38:57.544238 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:57.544198    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:57.544719 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:57.544251    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:57.556462 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:57.556441    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:38:58.269615 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:38:58.269587    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:39:18.408658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:18.408624    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:39:23.462091 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:23.462058    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4_5e63d677-b2fb-4b10-8214-d55daba0e041/payload-processing/0.log"
Sep 08 15:39:28.618880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.618844    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:39:28.623696 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.623673    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.628752 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.628731    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-6f830d\""
Sep 08 15:39:28.628887 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.628771    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-6f830d-dockercfg-t4lw5\""
Sep 08 15:39:28.638028 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.638001    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:39:28.723141 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.723110    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h959\" (UniqueName: \"kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.723417 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.723266    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.824075 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.824022    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.824256 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.824137    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h959\" (UniqueName: \"kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.824759 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.824740    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.833760 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.833725    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h959\" (UniqueName: \"kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959\") pod \"payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:28.934087 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:28.934011    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:29.206022 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:29.205948    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:39:29.209908 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:39:29.209886    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeef0e7b_7c5c_43b8_b0b1_6960d58a2836.slice/crio-d81e65e9bb309eff1bb369f8fdb02fec288320a0c6ae45732110547bb6ea8db2 WatchSource:0}: Error finding container d81e65e9bb309eff1bb369f8fdb02fec288320a0c6ae45732110547bb6ea8db2: Status 404 returned error can't find the container with id d81e65e9bb309eff1bb369f8fdb02fec288320a0c6ae45732110547bb6ea8db2
Sep 08 15:39:29.425203 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:29.425173    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" event={"ID":"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836","Type":"ContainerStarted","Data":"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"}
Sep 08 15:39:29.425383 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:29.425225    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" event={"ID":"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836","Type":"ContainerStarted","Data":"d81e65e9bb309eff1bb369f8fdb02fec288320a0c6ae45732110547bb6ea8db2"}
Sep 08 15:39:29.425383 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:29.425255    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:29.447826 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:29.447769    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" podStartSLOduration=1.44775154 podStartE2EDuration="1.44775154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:39:29.443214198 +0000 UTC m=+1611.286011421" watchObservedRunningTime="2026-09-08 15:39:29.44775154 +0000 UTC m=+1611.290548768"
Sep 08 15:39:36.808330 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:36.808230    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:39:40.434767 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:40.434735    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:58.718031 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:58.718000    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:39:58.718595 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:58.718201    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" podUID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" containerName="payload-processing" containerID="cri-o://cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092" gracePeriod=30
Sep 08 15:39:58.969048 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:58.968994    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:59.094556 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.094525    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h959\" (UniqueName: \"kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959\") pod \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") "
Sep 08 15:39:59.094730 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.094617    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume\") pod \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\" (UID: \"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836\") "
Sep 08 15:39:59.094976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.094952    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" (UID: "eeef0e7b-7c5c-43b8-b0b1-6960d58a2836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:39:59.096672 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.096650    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959" (OuterVolumeSpecName: "kube-api-access-9h959") pod "eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" (UID: "eeef0e7b-7c5c-43b8-b0b1-6960d58a2836"). InnerVolumeSpecName "kube-api-access-9h959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:39:59.195939 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.195912    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:39:59.195939 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.195934    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h959\" (UniqueName: \"kubernetes.io/projected/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836-kube-api-access-9h959\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:39:59.596731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.596694    2568 generic.go:356] "Generic (PLEG): container finished" podID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" containerID="cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092" exitCode=0
Sep 08 15:39:59.596918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.596754    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"
Sep 08 15:39:59.596918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.596755    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" event={"ID":"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836","Type":"ContainerDied","Data":"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"}
Sep 08 15:39:59.596918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.596880    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff" event={"ID":"eeef0e7b-7c5c-43b8-b0b1-6960d58a2836","Type":"ContainerDied","Data":"d81e65e9bb309eff1bb369f8fdb02fec288320a0c6ae45732110547bb6ea8db2"}
Sep 08 15:39:59.596918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.596910    2568 scope.go:117] "RemoveContainer" containerID="cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"
Sep 08 15:39:59.606495 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.606407    2568 scope.go:117] "RemoveContainer" containerID="cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"
Sep 08 15:39:59.606785 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:39:59.606756    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092\": container with ID starting with cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092 not found: ID does not exist" containerID="cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"
Sep 08 15:39:59.606884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.606796    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092"} err="failed to get container status \"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092\": rpc error: code = NotFound desc = could not find container \"cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092\": container with ID starting with cd93e0f5ce4b502fa320157256978943b718d58037a3ba0c4c7b47dd5feca092 not found: ID does not exist"
Sep 08 15:39:59.643669 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.643644    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:39:59.646784 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:39:59.646764    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-6f830d-5fb9fffddf-959ff"]
Sep 08 15:40:00.675853 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:00.675820    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" path="/var/lib/kubelet/pods/eeef0e7b-7c5c-43b8-b0b1-6960d58a2836/volumes"
Sep 08 15:40:04.409419 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:04.409385    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:40:24.716826 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:24.716793    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:40:30.211596 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:30.211561    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:40:34.600840 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:34.600806    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:40:45.231411 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.231363    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" podUID="04592b51-dfee-49f1-bb96-c0b62717aa7c" containerName="main" containerID="cri-o://76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee" gracePeriod=45
Sep 08 15:40:45.486981 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.486922    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:40:45.589721 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589685    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.589721 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589722    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.589929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589745    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.589929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589796    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cjzf\" (UniqueName: \"kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.589929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589854    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.589929 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589892    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm\") pod \"04592b51-dfee-49f1-bb96-c0b62717aa7c\" (UID: \"04592b51-dfee-49f1-bb96-c0b62717aa7c\") "
Sep 08 15:40:45.590087 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.589979    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:40:45.590087 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.590003    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:40:45.590152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.590109    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home" (OuterVolumeSpecName: "home") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:40:45.590191 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.590168    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.590191 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.590181    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.592143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.592106    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:40:45.592143 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.592121    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm" (OuterVolumeSpecName: "dshm") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:40:45.592328 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.592175    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf" (OuterVolumeSpecName: "kube-api-access-9cjzf") pod "04592b51-dfee-49f1-bb96-c0b62717aa7c" (UID: "04592b51-dfee-49f1-bb96-c0b62717aa7c"). InnerVolumeSpecName "kube-api-access-9cjzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:40:45.690858 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.690834    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04592b51-dfee-49f1-bb96-c0b62717aa7c-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.690858 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.690858    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.691023 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.690870    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04592b51-dfee-49f1-bb96-c0b62717aa7c-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.691023 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.690878    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjzf\" (UniqueName: \"kubernetes.io/projected/04592b51-dfee-49f1-bb96-c0b62717aa7c-kube-api-access-9cjzf\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:40:45.848846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.848763    2568 generic.go:356] "Generic (PLEG): container finished" podID="04592b51-dfee-49f1-bb96-c0b62717aa7c" containerID="76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee" exitCode=0
Sep 08 15:40:45.848846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.848805    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"
Sep 08 15:40:45.848846 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.848819    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" event={"ID":"04592b51-dfee-49f1-bb96-c0b62717aa7c","Type":"ContainerDied","Data":"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"}
Sep 08 15:40:45.849089 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.848860    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn" event={"ID":"04592b51-dfee-49f1-bb96-c0b62717aa7c","Type":"ContainerDied","Data":"8b84543a1c011da6c4ba6d228e9a78f96fc5029459a0c03b59e75161d16b37e7"}
Sep 08 15:40:45.849089 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.848882    2568 scope.go:117] "RemoveContainer" containerID="76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"
Sep 08 15:40:45.858411 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.858392    2568 scope.go:117] "RemoveContainer" containerID="76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"
Sep 08 15:40:45.858657 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:40:45.858633    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee\": container with ID starting with 76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee not found: ID does not exist" containerID="76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"
Sep 08 15:40:45.858755 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.858663    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee"} err="failed to get container status \"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee\": rpc error: code = NotFound desc = could not find container \"76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee\": container with ID starting with 76f2cc361e0d4a90a39bec579ab95a8bc820a969c797329dfdc7eeaffa7579ee not found: ID does not exist"
Sep 08 15:40:45.869246 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.869223    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:40:45.872790 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:45.872769    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-014f36/ipp-route-014f36-kserve-79fd5fd8cf-5dpgn"]
Sep 08 15:40:46.675888 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:40:46.675857    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04592b51-dfee-49f1-bb96-c0b62717aa7c" path="/var/lib/kubelet/pods/04592b51-dfee-49f1-bb96-c0b62717aa7c/volumes"
Sep 08 15:41:24.031202 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.031169    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:41:24.031676 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.031399    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" podUID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" containerName="maas-api" containerID="cri-o://7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd" gracePeriod=30
Sep 08 15:41:24.086577 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.086542    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:41:24.086816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.086788    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" podUID="5e63d677-b2fb-4b10-8214-d55daba0e041" containerName="payload-processing" containerID="cri-o://9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea" gracePeriod=30
Sep 08 15:41:24.334879 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.334852    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:41:24.370216 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.370189    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:41:24.412856 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.412813    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs\") pod \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") "
Sep 08 15:41:24.413039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.412878    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume\") pod \"5e63d677-b2fb-4b10-8214-d55daba0e041\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") "
Sep 08 15:41:24.413039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.412910    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca\") pod \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") "
Sep 08 15:41:24.413039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.412941    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wt4k\" (UniqueName: \"kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k\") pod \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") "
Sep 08 15:41:24.413039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.412988    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls\") pod \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\" (UID: \"c6064711-6dc9-4d6e-9642-a789f1cde8ec\") "
Sep 08 15:41:24.413039 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.413034    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgr68\" (UniqueName: \"kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68\") pod \"5e63d677-b2fb-4b10-8214-d55daba0e041\" (UID: \"5e63d677-b2fb-4b10-8214-d55daba0e041\") "
Sep 08 15:41:24.413364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.413317    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e63d677-b2fb-4b10-8214-d55daba0e041" (UID: "5e63d677-b2fb-4b10-8214-d55daba0e041"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:24.413364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.413338    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6064711-6dc9-4d6e-9642-a789f1cde8ec" (UID: "c6064711-6dc9-4d6e-9642-a789f1cde8ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:24.415155 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.415120    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c6064711-6dc9-4d6e-9642-a789f1cde8ec" (UID: "c6064711-6dc9-4d6e-9642-a789f1cde8ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:24.415155 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.415120    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k" (OuterVolumeSpecName: "kube-api-access-7wt4k") pod "c6064711-6dc9-4d6e-9642-a789f1cde8ec" (UID: "c6064711-6dc9-4d6e-9642-a789f1cde8ec"). InnerVolumeSpecName "kube-api-access-7wt4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:24.415552 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.415524    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6064711-6dc9-4d6e-9642-a789f1cde8ec" (UID: "c6064711-6dc9-4d6e-9642-a789f1cde8ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:24.415608 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.415594    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68" (OuterVolumeSpecName: "kube-api-access-tgr68") pod "5e63d677-b2fb-4b10-8214-d55daba0e041" (UID: "5e63d677-b2fb-4b10-8214-d55daba0e041"). InnerVolumeSpecName "kube-api-access-tgr68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:24.514189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514154    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgr68\" (UniqueName: \"kubernetes.io/projected/5e63d677-b2fb-4b10-8214-d55daba0e041-kube-api-access-tgr68\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:24.514189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514184    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:24.514189 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514194    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e63d677-b2fb-4b10-8214-d55daba0e041-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:24.514467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514203    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6064711-6dc9-4d6e-9642-a789f1cde8ec-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:24.514467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514212    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wt4k\" (UniqueName: \"kubernetes.io/projected/c6064711-6dc9-4d6e-9642-a789f1cde8ec-kube-api-access-7wt4k\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:24.514467 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:24.514220    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6064711-6dc9-4d6e-9642-a789f1cde8ec-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:25.062824 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.062793    2568 generic.go:356] "Generic (PLEG): container finished" podID="5e63d677-b2fb-4b10-8214-d55daba0e041" containerID="9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea" exitCode=0
Sep 08 15:41:25.063204 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.062848    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"
Sep 08 15:41:25.063204 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.062879    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" event={"ID":"5e63d677-b2fb-4b10-8214-d55daba0e041","Type":"ContainerDied","Data":"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"}
Sep 08 15:41:25.063204 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.062944    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4" event={"ID":"5e63d677-b2fb-4b10-8214-d55daba0e041","Type":"ContainerDied","Data":"f053084c1dce077cd63a4da643dab8ce80a69424880b1c91a11277ea9d1a3663"}
Sep 08 15:41:25.063204 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.062980    2568 scope.go:117] "RemoveContainer" containerID="9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"
Sep 08 15:41:25.064398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.064378    2568 generic.go:356] "Generic (PLEG): container finished" podID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" containerID="7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd" exitCode=0
Sep 08 15:41:25.064500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.064418    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" event={"ID":"c6064711-6dc9-4d6e-9642-a789f1cde8ec","Type":"ContainerDied","Data":"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"}
Sep 08 15:41:25.064500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.064437    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"
Sep 08 15:41:25.064500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.064450    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx" event={"ID":"c6064711-6dc9-4d6e-9642-a789f1cde8ec","Type":"ContainerDied","Data":"fe65fdd4e8025b1a73f1861ebea725c9195bc2914e4d1fb5d5656f40939135ff"}
Sep 08 15:41:25.072534 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.072517    2568 scope.go:117] "RemoveContainer" containerID="9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"
Sep 08 15:41:25.072816 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:41:25.072790    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea\": container with ID starting with 9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea not found: ID does not exist" containerID="9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"
Sep 08 15:41:25.072903 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.072828    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea"} err="failed to get container status \"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea\": rpc error: code = NotFound desc = could not find container \"9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea\": container with ID starting with 9933a6c7e12444086a3e25a506f3f15dd5fb52218b39639b36c62a68aef9fdea not found: ID does not exist"
Sep 08 15:41:25.072903 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.072851    2568 scope.go:117] "RemoveContainer" containerID="7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"
Sep 08 15:41:25.081081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.081063    2568 scope.go:117] "RemoveContainer" containerID="7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"
Sep 08 15:41:25.081358 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:41:25.081340    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd\": container with ID starting with 7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd not found: ID does not exist" containerID="7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"
Sep 08 15:41:25.081428 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.081364    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd"} err="failed to get container status \"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd\": rpc error: code = NotFound desc = could not find container \"7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd\": container with ID starting with 7d1afe60d925d537fd9519856224d471d059a7963dfe4e7323c2625d772132dd not found: ID does not exist"
Sep 08 15:41:25.090725 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.090697    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:41:25.104678 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.104654    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-30cc8d-6d84b5475b-xhslx"]
Sep 08 15:41:25.130111 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.130082    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:41:25.135103 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:25.135073    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-30cc8d-75748f4f-72nx4"]
Sep 08 15:41:26.676937 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:26.676901    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e63d677-b2fb-4b10-8214-d55daba0e041" path="/var/lib/kubelet/pods/5e63d677-b2fb-4b10-8214-d55daba0e041/volumes"
Sep 08 15:41:26.677336 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:26.677315    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" path="/var/lib/kubelet/pods/c6064711-6dc9-4d6e-9642-a789f1cde8ec/volumes"
Sep 08 15:41:28.403735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:28.403704    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:41:42.087532 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.087498    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:41:42.087962 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.087756    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" podUID="951300b2-7a58-407d-bbca-2e8060822b96" containerName="maas-api" containerID="cri-o://f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9" gracePeriod=30
Sep 08 15:41:42.156205 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.156174    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:41:42.156518 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.156470    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" podUID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" containerName="payload-pre-processing" containerID="cri-o://0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389" gracePeriod=30
Sep 08 15:41:42.394658 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.394628    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:41:42.425533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.425510    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:41:42.454251 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454225    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume\") pod \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") "
Sep 08 15:41:42.454404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454259    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls\") pod \"951300b2-7a58-407d-bbca-2e8060822b96\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") "
Sep 08 15:41:42.454404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454320    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca\") pod \"951300b2-7a58-407d-bbca-2e8060822b96\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") "
Sep 08 15:41:42.454404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454358    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w2xg\" (UniqueName: \"kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg\") pod \"951300b2-7a58-407d-bbca-2e8060822b96\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") "
Sep 08 15:41:42.454404 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454381    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs\") pod \"951300b2-7a58-407d-bbca-2e8060822b96\" (UID: \"951300b2-7a58-407d-bbca-2e8060822b96\") "
Sep 08 15:41:42.454629 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454424    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th87v\" (UniqueName: \"kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v\") pod \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\" (UID: \"938315a2-b1e8-4e7f-98bd-0d615d857c5a\") "
Sep 08 15:41:42.454694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454658    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "938315a2-b1e8-4e7f-98bd-0d615d857c5a" (UID: "938315a2-b1e8-4e7f-98bd-0d615d857c5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:42.454694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.454679    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "951300b2-7a58-407d-bbca-2e8060822b96" (UID: "951300b2-7a58-407d-bbca-2e8060822b96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:41:42.456530 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.456496    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "951300b2-7a58-407d-bbca-2e8060822b96" (UID: "951300b2-7a58-407d-bbca-2e8060822b96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:42.456666 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.456581    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v" (OuterVolumeSpecName: "kube-api-access-th87v") pod "938315a2-b1e8-4e7f-98bd-0d615d857c5a" (UID: "938315a2-b1e8-4e7f-98bd-0d615d857c5a"). InnerVolumeSpecName "kube-api-access-th87v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:42.456923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.456902    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "951300b2-7a58-407d-bbca-2e8060822b96" (UID: "951300b2-7a58-407d-bbca-2e8060822b96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:41:42.456923 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.456918    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg" (OuterVolumeSpecName: "kube-api-access-5w2xg") pod "951300b2-7a58-407d-bbca-2e8060822b96" (UID: "951300b2-7a58-407d-bbca-2e8060822b96"). InnerVolumeSpecName "kube-api-access-5w2xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:41:42.555514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555469    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w2xg\" (UniqueName: \"kubernetes.io/projected/951300b2-7a58-407d-bbca-2e8060822b96-kube-api-access-5w2xg\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.555514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555507    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.555514 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555522    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th87v\" (UniqueName: \"kubernetes.io/projected/938315a2-b1e8-4e7f-98bd-0d615d857c5a-kube-api-access-th87v\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.555735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555537    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/938315a2-b1e8-4e7f-98bd-0d615d857c5a-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.555735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555548    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951300b2-7a58-407d-bbca-2e8060822b96-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:42.555735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:42.555560    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951300b2-7a58-407d-bbca-2e8060822b96-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:41:43.173667 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.173636    2568 generic.go:356] "Generic (PLEG): container finished" podID="951300b2-7a58-407d-bbca-2e8060822b96" containerID="f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9" exitCode=0
Sep 08 15:41:43.174115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.173679    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"
Sep 08 15:41:43.174115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.173732    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" event={"ID":"951300b2-7a58-407d-bbca-2e8060822b96","Type":"ContainerDied","Data":"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"}
Sep 08 15:41:43.174115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.173789    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr" event={"ID":"951300b2-7a58-407d-bbca-2e8060822b96","Type":"ContainerDied","Data":"8e39a5d309eb185d0a14d2f1055168f68cca5e658e9f2f45027f8a8f25b46794"}
Sep 08 15:41:43.174115 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.173821    2568 scope.go:117] "RemoveContainer" containerID="f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"
Sep 08 15:41:43.175445 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.175421    2568 generic.go:356] "Generic (PLEG): container finished" podID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" containerID="0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389" exitCode=0
Sep 08 15:41:43.175588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.175452    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"
Sep 08 15:41:43.175588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.175495    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" event={"ID":"938315a2-b1e8-4e7f-98bd-0d615d857c5a","Type":"ContainerDied","Data":"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"}
Sep 08 15:41:43.175588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.175572    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz" event={"ID":"938315a2-b1e8-4e7f-98bd-0d615d857c5a","Type":"ContainerDied","Data":"2b0a33ebc56d9046dda68fca9326d87a23e2228d9a46fb14fd618f9a0f2d5e78"}
Sep 08 15:41:43.183524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.183507    2568 scope.go:117] "RemoveContainer" containerID="f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"
Sep 08 15:41:43.183787 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:41:43.183765    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9\": container with ID starting with f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9 not found: ID does not exist" containerID="f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"
Sep 08 15:41:43.183869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.183797    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9"} err="failed to get container status \"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9\": rpc error: code = NotFound desc = could not find container \"f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9\": container with ID starting with f9197e258331dc8242282cb7d91c3ff0d72e687889314117eab930275bedd5f9 not found: ID does not exist"
Sep 08 15:41:43.183869 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.183820    2568 scope.go:117] "RemoveContainer" containerID="0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"
Sep 08 15:41:43.192078 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.192060    2568 scope.go:117] "RemoveContainer" containerID="0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"
Sep 08 15:41:43.192348 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:41:43.192329    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389\": container with ID starting with 0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389 not found: ID does not exist" containerID="0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"
Sep 08 15:41:43.192418 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.192355    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389"} err="failed to get container status \"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389\": rpc error: code = NotFound desc = could not find container \"0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389\": container with ID starting with 0fbe0ee81de7a9057bb397f90c902ac759fb674dcd430464c9a3fee628c76389 not found: ID does not exist"
Sep 08 15:41:43.196549 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.196527    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:41:43.201194 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.201175    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fc867c-6b5d84dbc4-76qcz"]
Sep 08 15:41:43.210969 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.210938    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:41:43.217054 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:43.217035    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-fc867c-5ddd86b4db-kptsr"]
Sep 08 15:41:44.675711 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:44.675681    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" path="/var/lib/kubelet/pods/938315a2-b1e8-4e7f-98bd-0d615d857c5a/volumes"
Sep 08 15:41:44.676081 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:44.676054    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951300b2-7a58-407d-bbca-2e8060822b96" path="/var/lib/kubelet/pods/951300b2-7a58-407d-bbca-2e8060822b96/volumes"
Sep 08 15:41:45.899571 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:45.899535    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:41:59.753517 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:59.753484    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:41:59.754105 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:41:59.753721    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" podUID="30b7d815-292a-4009-9de2-59c7113ad9b1" containerName="payload-pre-processing" containerID="cri-o://43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10" gracePeriod=30
Sep 08 15:42:00.005962 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.005894    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:42:00.102640 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.102614    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume\") pod \"30b7d815-292a-4009-9de2-59c7113ad9b1\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") "
Sep 08 15:42:00.102816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.102652    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6cwd\" (UniqueName: \"kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd\") pod \"30b7d815-292a-4009-9de2-59c7113ad9b1\" (UID: \"30b7d815-292a-4009-9de2-59c7113ad9b1\") "
Sep 08 15:42:00.103027 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.103002    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30b7d815-292a-4009-9de2-59c7113ad9b1" (UID: "30b7d815-292a-4009-9de2-59c7113ad9b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:42:00.104788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.104764    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd" (OuterVolumeSpecName: "kube-api-access-l6cwd") pod "30b7d815-292a-4009-9de2-59c7113ad9b1" (UID: "30b7d815-292a-4009-9de2-59c7113ad9b1"). InnerVolumeSpecName "kube-api-access-l6cwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:42:00.203652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.203625    2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b7d815-292a-4009-9de2-59c7113ad9b1-plugins-config-volume\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:42:00.203652 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.203647    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6cwd\" (UniqueName: \"kubernetes.io/projected/30b7d815-292a-4009-9de2-59c7113ad9b1-kube-api-access-l6cwd\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:42:00.270896 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.270819    2568 generic.go:356] "Generic (PLEG): container finished" podID="30b7d815-292a-4009-9de2-59c7113ad9b1" containerID="43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10" exitCode=0
Sep 08 15:42:00.270896 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.270862    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"
Sep 08 15:42:00.271066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.270902    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" event={"ID":"30b7d815-292a-4009-9de2-59c7113ad9b1","Type":"ContainerDied","Data":"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"}
Sep 08 15:42:00.271066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.270955    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77" event={"ID":"30b7d815-292a-4009-9de2-59c7113ad9b1","Type":"ContainerDied","Data":"2c481dbcba19bc922c042fb91c0afd00cbdb6f9b1732362938d492a6f5111c6b"}
Sep 08 15:42:00.271066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.270980    2568 scope.go:117] "RemoveContainer" containerID="43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"
Sep 08 15:42:00.280561 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.280543    2568 scope.go:117] "RemoveContainer" containerID="43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"
Sep 08 15:42:00.280874 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:42:00.280855    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10\": container with ID starting with 43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10 not found: ID does not exist" containerID="43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"
Sep 08 15:42:00.280977 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.280879    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10"} err="failed to get container status \"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10\": rpc error: code = NotFound desc = could not find container \"43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10\": container with ID starting with 43af0240b65934827969247a9b50c8e3b33ea600a4145bf009b3fc9cd8176c10 not found: ID does not exist"
Sep 08 15:42:00.302105 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.302080    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:42:00.315171 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.309668    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-565cd7-76cd6f664b-gjf77"]
Sep 08 15:42:00.675557 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:00.675527    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b7d815-292a-4009-9de2-59c7113ad9b1" path="/var/lib/kubelet/pods/30b7d815-292a-4009-9de2-59c7113ad9b1/volumes"
Sep 08 15:42:03.895512 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:03.895476    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:42:11.204037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:11.204001    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:42:38.762965 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:38.762935    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:42:38.771031 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:38.771009    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:42:39.235821 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:42:39.235785    2568 scope.go:117] "RemoveContainer" containerID="5468a6ea77798c30646b98e98f57fa572095bb9718bda4875f8b70fbce54db9c"
Sep 08 15:43:30.661156 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:43:30.661116    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:43:55.800424 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:43:55.800380    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:44:04.007442 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:44:04.007358    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:44:11.610834 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:44:11.610797    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:44:30.216500 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:44:30.216459    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:45:57.412450 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:45:57.412371    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:05.608526 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:05.608489    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:11.220580 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:11.220542    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:24.611669 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.611636    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.611975    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951300b2-7a58-407d-bbca-2e8060822b96" containerName="maas-api"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.611986    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="951300b2-7a58-407d-bbca-2e8060822b96" containerName="maas-api"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.611996    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04592b51-dfee-49f1-bb96-c0b62717aa7c" containerName="main"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612001    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="04592b51-dfee-49f1-bb96-c0b62717aa7c" containerName="main"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612010    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" containerName="payload-pre-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612016    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" containerName="payload-pre-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612025    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" containerName="maas-api"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612031    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" containerName="maas-api"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612040    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" containerName="payload-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612045    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" containerName="payload-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612054    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b7d815-292a-4009-9de2-59c7113ad9b1" containerName="payload-pre-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612059    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b7d815-292a-4009-9de2-59c7113ad9b1" containerName="payload-pre-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612065    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e63d677-b2fb-4b10-8214-d55daba0e041" containerName="payload-processing"
Sep 08 15:46:24.612055 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612070    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e63d677-b2fb-4b10-8214-d55daba0e041" containerName="payload-processing"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612153    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="938315a2-b1e8-4e7f-98bd-0d615d857c5a" containerName="payload-pre-processing"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612162    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeef0e7b-7c5c-43b8-b0b1-6960d58a2836" containerName="payload-processing"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612169    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6064711-6dc9-4d6e-9642-a789f1cde8ec" containerName="maas-api"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612175    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="04592b51-dfee-49f1-bb96-c0b62717aa7c" containerName="main"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612180    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e63d677-b2fb-4b10-8214-d55daba0e041" containerName="payload-processing"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612186    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b7d815-292a-4009-9de2-59c7113ad9b1" containerName="payload-pre-processing"
Sep 08 15:46:24.612602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.612191    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="951300b2-7a58-407d-bbca-2e8060822b96" containerName="maas-api"
Sep 08 15:46:24.615144 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.615127    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.617457 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.617436    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-360f3caf-kserve-self-signed-certs\""
Sep 08 15:46:24.630694 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.630674    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:46:24.753797 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753761    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.753797 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753795    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.754047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753856    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.754047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753921    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hwm\" (UniqueName: \"kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.754047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753965    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.754047 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.753987    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855217    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855249 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855248    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855280    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855417    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hwm\" (UniqueName: \"kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855493 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855471    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855671 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855731 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855716    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855743    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.855796 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.855774    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.857465 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.857441    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.857932 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.857914    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.863721 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.863678    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hwm\" (UniqueName: \"kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm\") pod \"e2e-same-id-a-360f3caf-kserve-8859965f-788wg\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") " pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:24.924818 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:24.924791    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:25.200515 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:25.200487    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:46:25.202060 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:46:25.202035    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3671383_21ba_43ec_9b47_4467fe7756b3.slice/crio-f68e75b754135b7ac14bcaa39b7650d28d6da808cae3662571f9810dd77e6b5b WatchSource:0}: Error finding container f68e75b754135b7ac14bcaa39b7650d28d6da808cae3662571f9810dd77e6b5b: Status 404 returned error can't find the container with id f68e75b754135b7ac14bcaa39b7650d28d6da808cae3662571f9810dd77e6b5b
Sep 08 15:46:25.667057 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:25.667023    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" event={"ID":"c3671383-21ba-43ec-9b47-4467fe7756b3","Type":"ContainerStarted","Data":"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"}
Sep 08 15:46:25.667451 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:25.667070    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" event={"ID":"c3671383-21ba-43ec-9b47-4467fe7756b3","Type":"ContainerStarted","Data":"f68e75b754135b7ac14bcaa39b7650d28d6da808cae3662571f9810dd77e6b5b"}
Sep 08 15:46:25.687119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:25.687071    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" podStartSLOduration=1.687057059 podStartE2EDuration="1.687057059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:46:25.6846452 +0000 UTC m=+2027.527442423" watchObservedRunningTime="2026-09-08 15:46:25.687057059 +0000 UTC m=+2027.529854291"
Sep 08 15:46:27.401356 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:27.401314    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:34.925978 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:34.925942    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:34.926480 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:34.926001    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:34.938739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:34.938712    2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:35.728857 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:35.728827    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:46:37.221729 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:37.221683    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:45.250311 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:45.250267    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:46:49.202685 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:49.202647    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:46:59.009172 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:46:59.009141    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:00.269747 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.269701    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" podUID="c3671383-21ba-43ec-9b47-4467fe7756b3" containerName="main" containerID="cri-o://3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1" gracePeriod=45
Sep 08 15:47:00.511891 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.511866    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:47:00.539797 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539718    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.539797 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539777    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.539976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539812    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.539976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539860    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8hwm\" (UniqueName: \"kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.539976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539931    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.539976 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539963    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home\") pod \"c3671383-21ba-43ec-9b47-4467fe7756b3\" (UID: \"c3671383-21ba-43ec-9b47-4467fe7756b3\") "
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.539995    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.540126    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.540344    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home" (OuterVolumeSpecName: "home") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.540580    2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-home\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.540598    2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-model-cache\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.540182 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.540613    2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-tmp-dir\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.542210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.542179    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:00.542210 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.542184    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm" (OuterVolumeSpecName: "dshm") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:47:00.542382 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.542349    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm" (OuterVolumeSpecName: "kube-api-access-g8hwm") pod "c3671383-21ba-43ec-9b47-4467fe7756b3" (UID: "c3671383-21ba-43ec-9b47-4467fe7756b3"). InnerVolumeSpecName "kube-api-access-g8hwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:00.641715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.641680    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hwm\" (UniqueName: \"kubernetes.io/projected/c3671383-21ba-43ec-9b47-4467fe7756b3-kube-api-access-g8hwm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.641715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.641708    2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3671383-21ba-43ec-9b47-4467fe7756b3-tls-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.641715 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.641719    2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3671383-21ba-43ec-9b47-4467fe7756b3-dshm\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:00.848361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.848251    2568 generic.go:356] "Generic (PLEG): container finished" podID="c3671383-21ba-43ec-9b47-4467fe7756b3" containerID="3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1" exitCode=0
Sep 08 15:47:00.848361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.848312    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" event={"ID":"c3671383-21ba-43ec-9b47-4467fe7756b3","Type":"ContainerDied","Data":"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"}
Sep 08 15:47:00.848361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.848322    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"
Sep 08 15:47:00.848361 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.848353    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg" event={"ID":"c3671383-21ba-43ec-9b47-4467fe7756b3","Type":"ContainerDied","Data":"f68e75b754135b7ac14bcaa39b7650d28d6da808cae3662571f9810dd77e6b5b"}
Sep 08 15:47:00.848662 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.848376    2568 scope.go:117] "RemoveContainer" containerID="3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"
Sep 08 15:47:00.857398 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.857373    2568 scope.go:117] "RemoveContainer" containerID="3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"
Sep 08 15:47:00.857667 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:47:00.857644    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1\": container with ID starting with 3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1 not found: ID does not exist" containerID="3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"
Sep 08 15:47:00.857722 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.857675    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1"} err="failed to get container status \"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1\": rpc error: code = NotFound desc = could not find container \"3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1\": container with ID starting with 3763065190e05b62f611aeaadfd9bad7bb1e5794f8599a86bb7bf70ded57a3c1 not found: ID does not exist"
Sep 08 15:47:00.867633 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.867610    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:47:00.869605 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:00.869584    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-360f3caf-kserve-8859965f-788wg"]
Sep 08 15:47:02.675968 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:02.675934    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3671383-21ba-43ec-9b47-4467fe7756b3" path="/var/lib/kubelet/pods/c3671383-21ba-43ec-9b47-4467fe7756b3/volumes"
Sep 08 15:47:05.355524 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.355493    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:05.355919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.355852    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3671383-21ba-43ec-9b47-4467fe7756b3" containerName="main"
Sep 08 15:47:05.355919 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.355865    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3671383-21ba-43ec-9b47-4467fe7756b3" containerName="main"
Sep 08 15:47:05.355994 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.355933    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3671383-21ba-43ec-9b47-4467fe7756b3" containerName="main"
Sep 08 15:47:05.360332 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.360312    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.364739 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.364716    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4m2fn\""
Sep 08 15:47:05.364931 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.364917    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:47:05.365037 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.365016    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 15:47:05.365082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.365016    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:47:05.366158 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.366142    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:47:05.372251 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.372180    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:05.377267 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.377241    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.377411 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.377374    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.377479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.377442    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.377479 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.377473    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlj9z\" (UniqueName: \"kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.478576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.478544    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.478576 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.478580    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlj9z\" (UniqueName: \"kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.478820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.478602    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.478820 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.478644    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.479287 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.479262    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.481025 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.481002    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.481206 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.481185    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.487772 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.487749    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlj9z\" (UniqueName: \"kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z\") pod \"maas-api-8b8df4f6-g59hh\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") " pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.674147 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.674121    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:05.946680 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.946653    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:05.948265 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:47:05.948240    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305407b8_6270_4e22_8bcc_2a9ccd4de16c.slice/crio-0f6b78b72c48a688c088e8f44da0be0996ec4089496d174a83de91ba576b6b2a WatchSource:0}: Error finding container 0f6b78b72c48a688c088e8f44da0be0996ec4089496d174a83de91ba576b6b2a: Status 404 returned error can't find the container with id 0f6b78b72c48a688c088e8f44da0be0996ec4089496d174a83de91ba576b6b2a
Sep 08 15:47:05.949788 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:05.949765    2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:47:06.401911 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:06.401858    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:06.882759 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:06.882721    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" event={"ID":"305407b8-6270-4e22-8bcc-2a9ccd4de16c","Type":"ContainerStarted","Data":"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"}
Sep 08 15:47:06.882918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:06.882775    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" event={"ID":"305407b8-6270-4e22-8bcc-2a9ccd4de16c","Type":"ContainerStarted","Data":"0f6b78b72c48a688c088e8f44da0be0996ec4089496d174a83de91ba576b6b2a"}
Sep 08 15:47:06.882918 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:06.882879    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:06.900495 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:06.900452    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" podStartSLOduration=1.227245839 podStartE2EDuration="1.900438672s" totalImagesPullingTime="673.192833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:06.898741132 +0000 UTC m=+2068.741538357" watchObservedRunningTime="2026-09-08 15:47:06.900438672 +0000 UTC m=+2068.743235894"
Sep 08 15:47:12.895802 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:12.895767    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:13.183192 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.183110    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"]
Sep 08 15:47:13.186474 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.186456    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.198194 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.198163    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"]
Sep 08 15:47:13.248499 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.248470    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvrf\" (UniqueName: \"kubernetes.io/projected/164da088-9af4-4012-8126-63149f0c7e4a-kube-api-access-jdvrf\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.248625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.248512    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-maas-api-tls\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.248625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.248570    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-metrics-certs\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.248698 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.248635    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164da088-9af4-4012-8126-63149f0c7e4a-openshift-service-ca\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.349531 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.349494    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-metrics-certs\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.349665 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.349567    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164da088-9af4-4012-8126-63149f0c7e4a-openshift-service-ca\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.349665 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.349618    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvrf\" (UniqueName: \"kubernetes.io/projected/164da088-9af4-4012-8126-63149f0c7e4a-kube-api-access-jdvrf\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.349665 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.349660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-maas-api-tls\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.350392 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.350367    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/164da088-9af4-4012-8126-63149f0c7e4a-openshift-service-ca\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.352084 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.352060    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-maas-api-tls\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.352187 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.352099    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/164da088-9af4-4012-8126-63149f0c7e4a-metrics-certs\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.357998 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.357975    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvrf\" (UniqueName: \"kubernetes.io/projected/164da088-9af4-4012-8126-63149f0c7e4a-kube-api-access-jdvrf\") pod \"maas-api-59f6799c9b-qt4fs\" (UID: \"164da088-9af4-4012-8126-63149f0c7e4a\") " pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.496364 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.496262    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:13.502974 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.502949    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:13.768560 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.768451    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"]
Sep 08 15:47:13.770945 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:47:13.770922    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164da088_9af4_4012_8126_63149f0c7e4a.slice/crio-abf2e4da7bbc86784a854ccec2322e08209ed000851bc7e1b8367ac3121c4dc5 WatchSource:0}: Error finding container abf2e4da7bbc86784a854ccec2322e08209ed000851bc7e1b8367ac3121c4dc5: Status 404 returned error can't find the container with id abf2e4da7bbc86784a854ccec2322e08209ed000851bc7e1b8367ac3121c4dc5
Sep 08 15:47:13.928783 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:13.928746    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs" event={"ID":"164da088-9af4-4012-8126-63149f0c7e4a","Type":"ContainerStarted","Data":"abf2e4da7bbc86784a854ccec2322e08209ed000851bc7e1b8367ac3121c4dc5"}
Sep 08 15:47:14.935487 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:14.935451    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs" event={"ID":"164da088-9af4-4012-8126-63149f0c7e4a","Type":"ContainerStarted","Data":"becc8f55ef0bee89c97a06276cb526f3e730d78a0154a78f98f14d1238de0649"}
Sep 08 15:47:14.935884 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:14.935523    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:14.954003 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:14.953953    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs" podStartSLOduration=1.543630888 podStartE2EDuration="1.953939281s" totalImagesPullingTime="410.308393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:14.951830855 +0000 UTC m=+2076.794628079" watchObservedRunningTime="2026-09-08 15:47:14.953939281 +0000 UTC m=+2076.796736504"
Sep 08 15:47:20.946965 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:20.946932    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6799c9b-qt4fs"
Sep 08 15:47:20.994588 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:20.994557    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:20.994791 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:20.994749    2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" podUID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" containerName="maas-api" containerID="cri-o://ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6" gracePeriod=30
Sep 08 15:47:21.238414 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.238385    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:21.320573 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.320538    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlj9z\" (UniqueName: \"kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z\") pod \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") "
Sep 08 15:47:21.320751 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.320584    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs\") pod \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") "
Sep 08 15:47:21.320751 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.320635    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls\") pod \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") "
Sep 08 15:47:21.320751 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.320666    2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca\") pod \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\" (UID: \"305407b8-6270-4e22-8bcc-2a9ccd4de16c\") "
Sep 08 15:47:21.321059 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.321031    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "305407b8-6270-4e22-8bcc-2a9ccd4de16c" (UID: "305407b8-6270-4e22-8bcc-2a9ccd4de16c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:47:21.322844 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.322823    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "305407b8-6270-4e22-8bcc-2a9ccd4de16c" (UID: "305407b8-6270-4e22-8bcc-2a9ccd4de16c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:21.322937 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.322859    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "305407b8-6270-4e22-8bcc-2a9ccd4de16c" (UID: "305407b8-6270-4e22-8bcc-2a9ccd4de16c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:47:21.322937 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.322901    2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z" (OuterVolumeSpecName: "kube-api-access-tlj9z") pod "305407b8-6270-4e22-8bcc-2a9ccd4de16c" (UID: "305407b8-6270-4e22-8bcc-2a9ccd4de16c"). InnerVolumeSpecName "kube-api-access-tlj9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:47:21.422119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.422090    2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlj9z\" (UniqueName: \"kubernetes.io/projected/305407b8-6270-4e22-8bcc-2a9ccd4de16c-kube-api-access-tlj9z\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:21.422119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.422117    2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-metrics-certs\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:21.422119 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.422127    2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/305407b8-6270-4e22-8bcc-2a9ccd4de16c-maas-api-tls\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:21.422369 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.422134    2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/305407b8-6270-4e22-8bcc-2a9ccd4de16c-openshift-service-ca\") on node \"ip-10-0-134-180.ec2.internal\" DevicePath \"\""
Sep 08 15:47:21.973936 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.973901    2568 generic.go:356] "Generic (PLEG): container finished" podID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" containerID="ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6" exitCode=0
Sep 08 15:47:21.974415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.973953    2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"
Sep 08 15:47:21.974415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.973987    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" event={"ID":"305407b8-6270-4e22-8bcc-2a9ccd4de16c","Type":"ContainerDied","Data":"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"}
Sep 08 15:47:21.974415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.974040    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh" event={"ID":"305407b8-6270-4e22-8bcc-2a9ccd4de16c","Type":"ContainerDied","Data":"0f6b78b72c48a688c088e8f44da0be0996ec4089496d174a83de91ba576b6b2a"}
Sep 08 15:47:21.974415 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.974066    2568 scope.go:117] "RemoveContainer" containerID="ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"
Sep 08 15:47:21.985864 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.985845    2568 scope.go:117] "RemoveContainer" containerID="ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"
Sep 08 15:47:21.986172 ip-10-0-134-180 kubenswrapper[2568]: E0908 15:47:21.986150    2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6\": container with ID starting with ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6 not found: ID does not exist" containerID="ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"
Sep 08 15:47:21.986259 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.986180    2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6"} err="failed to get container status \"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6\": rpc error: code = NotFound desc = could not find container \"ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6\": container with ID starting with ea82997e3f30f702003ddb9db123e1acb1e3cda1c5ff3eb3018e3cb000d186c6 not found: ID does not exist"
Sep 08 15:47:21.996597 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.996576    2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:21.999795 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:21.999765    2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8b8df4f6-g59hh"]
Sep 08 15:47:22.675190 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:22.675157    2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" path="/var/lib/kubelet/pods/305407b8-6270-4e22-8bcc-2a9ccd4de16c/volumes"
Sep 08 15:47:23.308696 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:23.308662    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:30.403357 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:30.403323    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:34.411726 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:34.411689    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-bf5n7"]
Sep 08 15:47:38.238226 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:38.238193    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95999f5c-g47pq_5729da77-8fd7-4305-9730-ca1e4107c4fd/authorino/0.log"
Sep 08 15:47:38.798849 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:38.798820    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:47:38.807662 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:38.807639    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:47:44.257021 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:44.256991    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dbb6d945-qn7m8_a7d5fc44-22a8-41fd-9178-6429f54dd5f8/copy-manifests/0.log"
Sep 08 15:47:44.263064 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:44.263043    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dbb6d945-qn7m8_a7d5fc44-22a8-41fd-9178-6429f54dd5f8/manager/0.log"
Sep 08 15:47:44.693792 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:44.693765    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-59f6799c9b-qt4fs_164da088-9af4-4012-8126-63149f0c7e4a/maas-api/0.log"
Sep 08 15:47:45.782165 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:45.782129    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95999f5c-g47pq_5729da77-8fd7-4305-9730-ca1e4107c4fd/authorino/0.log"
Sep 08 15:47:46.133232 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:46.133203    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gkcnl_3e139d92-d50e-4c9e-a760-602dc4b51924/kuadrant-console-plugin/0.log"
Sep 08 15:47:46.382278 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:46.382248    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-bf5n7_00b13e06-4bff-46f3-afdd-d24884457cd9/limitador/0.log"
Sep 08 15:47:47.088538 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:47.088510    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565756cbb6-tp4jn_08e24b30-1d08-4f99-aaed-339b7fb97346/kube-auth-proxy/0.log"
Sep 08 15:47:47.666152 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:47.666123    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-845c498454-fdsb6_d127fd86-641c-4e8b-9dbb-a60dec4a0c5a/router/0.log"
Sep 08 15:47:48.345618 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:48.345588    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-p7kgf_d5829140-031c-4f1c-8406-1108063e1a34/main/0.log"
Sep 08 15:47:48.686184 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:48.686151    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6jd5s_d8e477ae-9b79-4fc9-a857-8ba500f885ba/main/0.log"
Sep 08 15:47:54.303650 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.303618    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m72hg/must-gather-jqs4x"]
Sep 08 15:47:54.304021 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.303952    2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" containerName="maas-api"
Sep 08 15:47:54.304021 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.303962    2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" containerName="maas-api"
Sep 08 15:47:54.304021 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.304020    2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="305407b8-6270-4e22-8bcc-2a9ccd4de16c" containerName="maas-api"
Sep 08 15:47:54.306857 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.306836    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.308816 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.308798    2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m72hg\"/\"default-dockercfg-hqgkw\""
Sep 08 15:47:54.308982 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.308967    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m72hg\"/\"kube-root-ca.crt\""
Sep 08 15:47:54.309221 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.309206    2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m72hg\"/\"openshift-service-ca.crt\""
Sep 08 15:47:54.321339 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.321285    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m72hg/must-gather-jqs4x"]
Sep 08 15:47:54.402653 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.402624    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrqs\" (UniqueName: \"kubernetes.io/projected/ace65868-b4c2-4ecf-91de-974078d611ca-kube-api-access-gvrqs\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.402793 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.402668    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ace65868-b4c2-4ecf-91de-974078d611ca-must-gather-output\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.503510 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.503480    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrqs\" (UniqueName: \"kubernetes.io/projected/ace65868-b4c2-4ecf-91de-974078d611ca-kube-api-access-gvrqs\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.503648 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.503525    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ace65868-b4c2-4ecf-91de-974078d611ca-must-gather-output\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.503813 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.503795    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ace65868-b4c2-4ecf-91de-974078d611ca-must-gather-output\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.511447 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.511423    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrqs\" (UniqueName: \"kubernetes.io/projected/ace65868-b4c2-4ecf-91de-974078d611ca-kube-api-access-gvrqs\") pod \"must-gather-jqs4x\" (UID: \"ace65868-b4c2-4ecf-91de-974078d611ca\") " pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.617475 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.617416    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m72hg/must-gather-jqs4x"
Sep 08 15:47:54.892941 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:54.892917    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m72hg/must-gather-jqs4x"]
Sep 08 15:47:54.895078 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:47:54.895052    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podace65868_b4c2_4ecf_91de_974078d611ca.slice/crio-372be54f3929030a0cf53b80276fb54145902bffafcd98c80be745ff5517722d WatchSource:0}: Error finding container 372be54f3929030a0cf53b80276fb54145902bffafcd98c80be745ff5517722d: Status 404 returned error can't find the container with id 372be54f3929030a0cf53b80276fb54145902bffafcd98c80be745ff5517722d
Sep 08 15:47:55.155644 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:55.155559    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m72hg/must-gather-jqs4x" event={"ID":"ace65868-b4c2-4ecf-91de-974078d611ca","Type":"ContainerStarted","Data":"372be54f3929030a0cf53b80276fb54145902bffafcd98c80be745ff5517722d"}
Sep 08 15:47:56.165435 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:56.165381    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m72hg/must-gather-jqs4x" event={"ID":"ace65868-b4c2-4ecf-91de-974078d611ca","Type":"ContainerStarted","Data":"dfe87213a3db1207c3dc044b8a798cf5e841db730dbb6e0657d72ee1be1c915e"}
Sep 08 15:47:56.165928 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:56.165453    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m72hg/must-gather-jqs4x" event={"ID":"ace65868-b4c2-4ecf-91de-974078d611ca","Type":"ContainerStarted","Data":"3390f8f2940517c4e34c147546cf351ca8201bf49b3629dcedf82c85f7334b1b"}
Sep 08 15:47:56.182684 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:56.182622    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m72hg/must-gather-jqs4x" podStartSLOduration=1.42969835 podStartE2EDuration="2.182586321s" totalImagesPullingTime="752.887971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:47:56.180544587 +0000 UTC m=+2118.023341812" watchObservedRunningTime="2026-09-08 15:47:56.182586321 +0000 UTC m=+2118.025383545"
Sep 08 15:47:57.232897 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:57.232864    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5zq8s_fa8efd27-0158-47b3-9e36-0ee192c74c98/global-pull-secret-syncer/0.log"
Sep 08 15:47:57.345985 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:57.345953    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmsgv_6dbae73f-a055-4d17-ad74-579f71d20b66/konnectivity-agent/0.log"
Sep 08 15:47:57.443382 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:47:57.443346    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-180.ec2.internal_cf1bd422ed1bc02488f8241ff80c9433/haproxy/0.log"
Sep 08 15:48:02.297265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:02.297226    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7d95999f5c-g47pq_5729da77-8fd7-4305-9730-ca1e4107c4fd/authorino/0.log"
Sep 08 15:48:02.406061 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:02.406027    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gkcnl_3e139d92-d50e-4c9e-a760-602dc4b51924/kuadrant-console-plugin/0.log"
Sep 08 15:48:02.479345 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:02.479317    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-bf5n7_00b13e06-4bff-46f3-afdd-d24884457cd9/limitador/0.log"
Sep 08 15:48:04.376367 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:04.376333    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-l6msf_d07b1e65-794c-4466-9cdc-3a2f5a796479/cluster-monitoring-operator/0.log"
Sep 08 15:48:04.603735 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:04.603675    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqhxl_e92d8002-ba45-476d-b4e5-8f97870d3756/node-exporter/0.log"
Sep 08 15:48:04.623926 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:04.623788    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqhxl_e92d8002-ba45-476d-b4e5-8f97870d3756/kube-rbac-proxy/0.log"
Sep 08 15:48:04.644758 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:04.644687    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqhxl_e92d8002-ba45-476d-b4e5-8f97870d3756/init-textfile/0.log"
Sep 08 15:48:05.053323 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:05.053224    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zgbrv_0ff40d75-0c7e-41ea-bc47-9a6bd824b54c/prometheus-operator-admission-webhook/0.log"
Sep 08 15:48:06.518248 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:06.518211    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-24dw5_849406e6-066e-4188-9f1c-062fd9d6bd5a/networking-console-plugin/0.log"
Sep 08 15:48:07.106800 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.106768    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/1.log"
Sep 08 15:48:07.117409 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.117374    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-fgn85_89520eb2-f313-47fa-abf2-61065322b60b/console-operator/2.log"
Sep 08 15:48:07.158386 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.158343    2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"]
Sep 08 15:48:07.166677 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.166645    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.168862 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.168831    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"]
Sep 08 15:48:07.242673 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.242351    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5v47\" (UniqueName: \"kubernetes.io/projected/2963e368-5b24-4636-ada2-b870c871a7ac-kube-api-access-c5v47\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.242673 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.242414    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-sys\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.242673 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.242476    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-proc\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.242673 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.242552    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-podres\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.242673 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.242584    2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-lib-modules\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343582    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-podres\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343625 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343623    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-lib-modules\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343660    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5v47\" (UniqueName: \"kubernetes.io/projected/2963e368-5b24-4636-ada2-b870c871a7ac-kube-api-access-c5v47\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343697    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-sys\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343741    2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-proc\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343803    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-lib-modules\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343812    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-podres\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343836    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-proc\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.343880 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.343820    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2963e368-5b24-4636-ada2-b870c871a7ac-sys\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.354707 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.354679    2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5v47\" (UniqueName: \"kubernetes.io/projected/2963e368-5b24-4636-ada2-b870c871a7ac-kube-api-access-c5v47\") pod \"perf-node-gather-daemonset-48546\" (UID: \"2963e368-5b24-4636-ada2-b870c871a7ac\") " pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.482629 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.482376    2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:07.775938 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:07.775911    2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"]
Sep 08 15:48:07.784304 ip-10-0-134-180 kubenswrapper[2568]: W0908 15:48:07.784249    2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2963e368_5b24_4636_ada2_b870c871a7ac.slice/crio-b39452f1701aee54cba9faaeb688017641026bf1069da16829b47098f34662d1 WatchSource:0}: Error finding container b39452f1701aee54cba9faaeb688017641026bf1069da16829b47098f34662d1: Status 404 returned error can't find the container with id b39452f1701aee54cba9faaeb688017641026bf1069da16829b47098f34662d1
Sep 08 15:48:08.266838 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:08.266795    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546" event={"ID":"2963e368-5b24-4636-ada2-b870c871a7ac","Type":"ContainerStarted","Data":"4362adaa0b13632fdd37c0287129cab317a3f8f31f5f8e3a669932376cbbd617"}
Sep 08 15:48:08.267066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:08.266862    2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546" event={"ID":"2963e368-5b24-4636-ada2-b870c871a7ac","Type":"ContainerStarted","Data":"b39452f1701aee54cba9faaeb688017641026bf1069da16829b47098f34662d1"}
Sep 08 15:48:08.267066 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:08.266991    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:08.284416 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:08.284363    2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546" podStartSLOduration=1.284344895 podStartE2EDuration="1.284344895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:48:08.284017439 +0000 UTC m=+2130.126814664" watchObservedRunningTime="2026-09-08 15:48:08.284344895 +0000 UTC m=+2130.127142120"
Sep 08 15:48:08.498214 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:08.498183    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-8xksq_6637b6b4-7c9a-46ae-a129-373cc09bd5d7/volume-data-source-validator/0.log"
Sep 08 15:48:09.651917 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:09.651884    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc2px_b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49/dns/0.log"
Sep 08 15:48:09.671616 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:09.671587    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rc2px_b8eee15c-7b5d-48d2-b5ca-5b1c2d6fdf49/kube-rbac-proxy/0.log"
Sep 08 15:48:09.742169 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:09.742139    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcdpg_d83812e0-af07-4572-8b7b-5f26596e433f/dns-node-resolver/0.log"
Sep 08 15:48:10.290265 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:10.290238    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44gr6_348f8ed7-b037-40a8-a964-500bd8d0520d/node-ca/0.log"
Sep 08 15:48:11.858809 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:11.858750    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565756cbb6-tp4jn_08e24b30-1d08-4f99-aaed-339b7fb97346/kube-auth-proxy/0.log"
Sep 08 15:48:12.017799 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:12.017769    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-845c498454-fdsb6_d127fd86-641c-4e8b-9dbb-a60dec4a0c5a/router/0.log"
Sep 08 15:48:12.907850 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:12.907792    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-njhks_04141987-e53e-4748-9ef5-897cc30217f7/serve-healthcheck-canary/0.log"
Sep 08 15:48:13.415141 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:13.415114    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fdg9x_125e8c76-09e0-466a-8518-6469c0b019eb/insights-operator/0.log"
Sep 08 15:48:13.417023 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:13.416992    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fdg9x_125e8c76-09e0-466a-8518-6469c0b019eb/insights-operator/1.log"
Sep 08 15:48:13.568828 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:13.568801    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzh5h_d80a4af0-164c-48f4-a132-19de631f60f4/kube-rbac-proxy/0.log"
Sep 08 15:48:13.591198 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:13.591168    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzh5h_d80a4af0-164c-48f4-a132-19de631f60f4/exporter/0.log"
Sep 08 15:48:13.615204 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:13.615179    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pzh5h_d80a4af0-164c-48f4-a132-19de631f60f4/extractor/0.log"
Sep 08 15:48:14.286921 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:14.286892    2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m72hg/perf-node-gather-daemonset-48546"
Sep 08 15:48:16.403999 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:16.403967    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dbb6d945-qn7m8_a7d5fc44-22a8-41fd-9178-6429f54dd5f8/manager/0.log"
Sep 08 15:48:16.424713 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:16.424685    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68dbb6d945-qn7m8_a7d5fc44-22a8-41fd-9178-6429f54dd5f8/copy-manifests/0.log"
Sep 08 15:48:17.760441 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:17.760401    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c78744898-gd52z_1d11219a-26ae-49c1-84d0-b879904036ef/manager/0.log"
Sep 08 15:48:18.298910 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:18.298883    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565756cbb6-tp4jn_08e24b30-1d08-4f99-aaed-339b7fb97346/kube-auth-proxy/0.log"
Sep 08 15:48:18.426188 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:18.426143    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-845c498454-fdsb6_d127fd86-641c-4e8b-9dbb-a60dec4a0c5a/router/0.log"
Sep 08 15:48:20.397631 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:20.397603    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-565756cbb6-tp4jn_08e24b30-1d08-4f99-aaed-339b7fb97346/kube-auth-proxy/0.log"
Sep 08 15:48:20.530748 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:20.530662    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-845c498454-fdsb6_d127fd86-641c-4e8b-9dbb-a60dec4a0c5a/router/0.log"
Sep 08 15:48:22.639602 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:22.639570    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ftz6x_6216abf5-f88e-427e-b963-b9d97787533a/migrator/0.log"
Sep 08 15:48:22.660737 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:22.660710    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-ftz6x_6216abf5-f88e-427e-b963-b9d97787533a/graceful-termination/0.log"
Sep 08 15:48:22.997420 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:22.997337    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fnjnx_8aac3e49-399c-4f97-a4b0-9343e12bc985/kube-storage-version-migrator-operator/1.log"
Sep 08 15:48:22.998082 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:22.998059    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-fnjnx_8aac3e49-399c-4f97-a4b0-9343e12bc985/kube-storage-version-migrator-operator/0.log"
Sep 08 15:48:24.080034 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.079955    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/kube-multus-additional-cni-plugins/0.log"
Sep 08 15:48:24.101499 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.101477    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/egress-router-binary-copy/0.log"
Sep 08 15:48:24.123882 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.123858    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/cni-plugins/0.log"
Sep 08 15:48:24.144845 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.144818    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/bond-cni-plugin/0.log"
Sep 08 15:48:24.167847 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.167805    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/routeoverride-cni/0.log"
Sep 08 15:48:24.188533 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.188508    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/whereabouts-cni-bincopy/0.log"
Sep 08 15:48:24.212574 ip-10-0-134-180 kubenswrapper[2568]: I0908 15:48:24.212542    2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-54rt7_78af1c5d-0364-4e1a-a601-b219b5fd12c8/whereabouts-cni/0.log"